Source-linked AI summary
HorizonBench: Long-Horizon Personalization with Evolving Preferences
Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz
TL;DR
Long-horizon personalization asks whether models can identify users’ current preferences after life events change previously stated values, a problem underserved by existing data and measurements. The paper generates six-month conversations from a structured mental state graph and builds HorizonBench with controlled preference evolution and diagnostic distractors. Across 25 frontier models, belief-update failures persist across context lengths and expression explicitness, making state tracking the primary reported bottleneck.
Problem
Long-horizon personalization lacks naturalistic long-term interactions with ground-truth provenance and controlled measurements of why models fail when preferences evolve.
Method
A structured mental state graph generates six-month conversations with provenance for preference changes, forming HorizonBench’s 4,245-item evaluation with controlled dimensions and pre-evolution distractors.
Results
52.8% is the best accuracy among 25 frontier models, while most score at or below the 20% chance baseline and evolved-preference errors consistently select pre-evolution values above chance.
Takeaways & Limitations
HorizonBench separates long-context retrieval from belief updating and identifies state tracking as the primary capability gap for long-horizon personalized agents.
Takeaways & Limitations
The benchmark uses synthetic LLM-generated conversations and multiple-choice recognition, so severity in naturalistic interactions and open-ended generation remains an open empirical question.
Abstract
from arXiv · showhide
User preferences evolve across months of interaction, and tracking them requires inferring when a stated preference has been changed by a subsequent life event. We define this problem as long-horizon personalization and observe that progress on it is limited by data availability and measurement, with no existing resource providing both naturalistic long-horizon interactions and the ground-truth provenance needed to diagnose why models fail. We introduce a data generator that produces conversations from a structured mental state graph, yielding ground-truth provenance for every preference change across 6-month timelines, and from it construct HorizonBench, a benchmark of 4,245 items from 360 simulated users with 6-month conversation histories averaging ~4,300 turns and ~163K tokens. HorizonBench provides a testbed for long-context modeling, memory-augmented architectures, theory-of-mind reasoning, and user modeling. Across 25 frontier models, the best model reaches 52.8% and most score at or below the 20% chance baseline. When these models err on evolved preferences, over a third of the time they select the user's originally stated value without tracking the updated user state. This belief-update failure persists across context lengths and expression explicitness levels, identifying state-tracking capability as the primary bottleneck for long-horizon personalization.
1 Introduction
Long-horizon personalization asks whether models can track preferences changed by later life events, but progress is constrained by missing ground truth and diagnostic controls. HorizonBench addresses these gaps with provenance-rich six-month timelines and controlled evaluation, revealing widespread belief-update failure.
- Problem: Long-horizon personalization identifies a user’s current preference after subsequent life events change previously stated preferences.Existing benchmarks largely model fixed traits rather than evolving user states.
- Problem: Progress is limited because real interactions do not reveal when or why preferences changed, while existing benchmarks lack controls for diagnosing model failures.Prior synthetic approaches also lack causal models of preference evolution.
- Benchmark: HorizonBench contains 4,245 items from 360 simulated users with six-month histories averaging ~4,300 turns and ~163K tokens.The benchmark uses pre-evolution values as hard-negative distractors and independently controls evolution status, expression explicitness, and context length.
- Results: 52.8% is the best accuracy among 25 frontier models, while most score at or below the 20% chance baseline.The benchmark therefore exposes substantial difficulty in long-horizon preference tracking.
- Contribution: A state-first generator and controlled benchmark separate memory retrieval from belief updating across evolving preferences.The resource provides ground-truth provenance for preference changes and supports targeted evaluation of both capabilities.
2 Related Work
Prior personalization and long-term memory benchmarks study retrieval, recall, and changing information, but generally lack controlled mechanisms for preference change. HorizonBench extends this work with structured, event-driven evolution and isolates belief-update failure.
- Prior benchmarks: Existing personalization benchmarks show difficulty with preference retrieval, long-context recall, and implicit preference expression.Several also report that dynamic preferences are harder than static ones.
- Prior benchmarks: Long-term memory benchmarks evaluate updating and extraction without modeling why information changes or isolating the sources of difficulty.They do not provide controlled experiments for separating retrieval from belief updating.
- This work: The data generator extends LLM agent simulation and lifelong personalization with a structured mental state graph, typed dependency edges, and full preference-evolution provenance.This design is motivated by evidence that state-level variation dominates trait-level variation in user behavior.
3 Problem Formulation
The paper formulates long-horizon personalization as identifying a user’s current preference from conversations generated by interacting stateful agents. A mental state graph links preferences and life events so changes and their provenance are known by construction.
- Task: Long-horizon personalization tracks how user preferences evolve across extended interactions and evaluates the current value from conversation history.Both user and assistant are represented as stateful agents with persona profiles and typed preference attributes.
- Representation: A mental state graph maintains each agent’s persona, social connections, life events, and preferences.The graph provides the structured state representation used to generate the interactions.
- Preference evolution: Typed dependency edges propagate life-event-driven changes among related preferences, while each change records its triggering event as provenance.An expression-tracking system distinguishes preference evolution from conversational expression.
4 HORIZONBENCH Data Generator
The data generator maintains structured agent states across six-month simulations, uses event-conditioned preference evolution with provenance, and produces controlled conversations for diagnosing retrieval and belief updating. Its configurable timelines, expression styles, and recall rules support targeted benchmark construction.
- Stateful generation: The generator’s structured mental state representation provides ground truth for every preference, its causal provenance, and its complete evolution history.These signals enable diagnostic evaluation and controlled experiments unavailable from observational data.
- Stateful generation: Conditioning every generation call on the mental state graph prevents persona drift and maintains coherent six-month interactions with isolable preference evolution.The graph persists personas, social connections, events, preferences, and provenance across the timeline.
- Event sampling: Events are sampled conditional on prior agent states and event history rather than independently, reducing unrealistic repetition.When stale preferences are selected for temporal recall, events are constrained to the relevant preference domain.
- Preference evolution: Typed dependency edges propagate a life event across related preferences, and with probability pevo = 0.15 one event changes 2–5 preferences simultaneously.The generator then creates an event-mention conversation without stating the new preference values directly.
- Preference evolution: A changed preference remains stale from its original expression, making its former value a valid hard-negative while requiring integration of the triggering life event.Retrieving the most recently expressed value therefore produces an outdated answer.
- Conversation generation: Outline-first generation controls which preferences surface, their order, and expression style before content is generated turn by turn.Expression variants rewrite preference-expressing turns into explicit or implicit forms while preserving the base conversation structure.
- Temporal recall: Preferences receive expression timestamps and become eligible for temporal recall after a default 30-day staleness threshold.Every recalled preference appears in the evaluation history, while evolved preferences also include the triggering event and a pre-evolution distractor.
- Configurability: Generator parameters such as timeline length, expression explicitness, evolution probability, staleness threshold, and demographics are configurable.This enables datasets tailored to specific research questions without relying on real user data.
5 HORIZONBENCH Benchmark
HORIZONBENCH evaluates whether models can identify a user's current preferences after those preferences evolve within long conversational histories. Its items use causally grounded timelines, hard-negative alternatives, and history filtering to test belief updates rather than surface matching.
- Benchmark task: HORIZONBENCH asks models to select the assistant response that best aligns with current preferences from a five-option question embedded in a long conversation history.Each item requires integrating prior preference expressions and subsequent changes across the timeline.
- Counterfactual evaluation: Preference-evolution items inject the original value as a hard-negative distractor alongside contextually appropriate counterfactual responses.The outdated option can remain plausible, so correct answers require tracking the updated preference state.
- History filtering: A strict five-model consensus filter retains items whose response options cannot be distinguished without the conversational history.The filter removes candidates that validation models can answer without seeing the full history.
- Dataset statistics: The benchmark is generated from 57,390 conversations across 360 users and 30 preference domains, producing 4,245 history-filtered items.The histories span six months and average roughly 4,300 turns per user.
- Benchmark validation: Human annotation produced 56–64% majority-vote accuracy, while models showed different errors by anchoring on outdated preferences.Human disagreements reflected ambiguity among plausible current interpretations rather than the model pattern of selecting pre-evolution values.
6 Experiments
Experiments show that models struggle more with evolved than static preferences and frequently select outdated values, even when the required information is present. Controlled analyses indicate that this failure persists across context lengths and counterfactual subtlety, implicating belief updating rather than retrieval alone.
- 6.1 Main Results: 52.8% was the highest accuracy among 25 frontier models, while 10 models fell below the 20% chance baseline on the five-option task.Position-debiased accuracy differed from raw accuracy by less than 1 percentage point for every model.
- 6.2 Belief-Update Failure: Over a third of wrong answers on evolved items selected the pre-evolution distractor, exceeding the 25% uniform-error baseline for all 25 models.The effect was significant at p < 0.001 and remained above 35% even for the best-performing model.
- 6.2 Belief-Update Failure: All 25 models performed worse on evolved than static preferences, with an average accuracy gap of −2.7 pp.The short-horizon variant widened the gap to −5.7 pp, despite using a shorter recall window.
- 6.3 Controlled Experiments: The controlled experiments varied context length, preference-expression explicitness, and counterfactual subtlety while measuring evolved-versus-static accuracy and distractor selection.These dimensions test whether belief-update failures can be explained by memory demand, surface cues, or item difficulty.
- 6.3 Controlled Experiments: Relaxing the consensus filter increased average accuracy by 11.9 pp, but the evolved-versus-static gap persisted and every model still selected the pre-evolution distractor above chance.At the relaxed threshold, the gap was −2.5 pp and distractor selection was 54.3%.
7 Conclusion
HorizonBench frames long-horizon personalization as tracking user preferences as they evolve across months, separating state tracking from retrieval. Across all 25 frontier models, evolved preferences expose a persistent belief-update failure.
- Conclusion: All 25 frontier models select pre-evolution distractors above chance on evolved preferences, regardless of context length, expression explicitness, or option subtlety.The evolved-vs-static gap persists even when histories are short enough that retrieval alone cannot close it.
- Conclusion: HorizonBench makes gains in retrieval and state tracking independently attributable for long-context, memory, and user-modeling research.The benchmark’s diagnostic design exposes whether a method improves retrieval, state tracking, or both.
- Conclusion: Explicit preference-state representations updated by life events and retrieval that surfaces both preferences and potentially changing events are proposed directions.
Limitations
The benchmark uses synthetic conversations to obtain experimental control that natural interaction data cannot provide, but its findings may not transfer directly to real long-horizon interactions. Its multiple-choice format also measures recognition rather than open-ended generation.
- Scope: Synthetic data provides known preference evolution, counterfactual distractors, and expression-explicitness variants unavailable in real interaction data.
- Scope: Whether the observed belief-update failure has the same severity in naturalistic long-horizon interactions remains an open empirical question.
- Evaluation format: The multiple-choice format evaluates preference tracking in a recognition setting, leaving open how belief-update failure appears in open-ended generation.
Ethics Statement
The supplied ethics-related passages describe benchmark scope, provenance, and evaluation context rather than a standalone ethics policy. They indicate that the benchmark uses synthetic data and evaluates model behavior on preference tracking.
- Context: Existing long-term memory benchmarks do not model why tracked information changes or provide controlled experiments isolating difficulty sources.
- Data and evaluation: The benchmark’s data-generation process uses synthetic conversational timelines with controlled preference evolution and counterfactual evaluation items.
- Evaluation: HorizonBench reports accuracy for 25 models across pooled data and subsets distinguished by generator and preference-evolution status.
- Data quality: Generator pass rates vary, with o3-generated items at 5.1% and Gemini-3-flash items at 17.6%.
D Human-in-the-Loop Iteration Details
Human annotation and iterative pipeline checks were used to validate benchmark quality, diagnose ambiguity and generation failures, and refine counterfactual construction. The pipeline generates controlled conversations and preference changes through structured prompts and explicit constraints.
- Human annotation: Three rounds of annotation used 10–20 trained annotators and three independent annotations per item to validate benchmark quality.
- Human annotation: In Round 2, at least one annotator was correct on 81% of items, while all annotators missed 19%, indicating a human difficulty floor.
- Model evaluation: Table 3 reports accuracy across 25 models, generator subsets, and evolved versus static preferences, with all models showing negative Δevo at p < 0.001.
- Pipeline improvements: Near-paraphrase distractors were strengthened so each option represented a meaningfully different attribute value.
- Pipeline improvements: Generation prompts were revised when responses failed to express the target preference or leaked literal numeric values.
- Disagreement analysis: Subjective attributes such as positivity level, strategic depth, and collaboration style produced high annotator disagreement.
- Disagreement analysis: Human disagreements reflected plausible alternative interpretations, including different positive tones, rather than being treated as annotation noise.
- Conversation generation: Conversation generation first plans an outline and then fills it with dialogue under verbal, behavioral, and other expression modes.
F Per-Model Controlled Experiment Results
Tables 6–7 report per-model belief-update metrics across neutral, short-horizon, explicit-rewrite, and implicit-rewrite conditions; Table 8 tests low-subtlety items. These results compare overall, static, and evolved-item accuracy, gaps, and pre-evolution distractor selection across controlled settings.
- Per-model metrics: Tables 6–7 report overall accuracy, static and evolved accuracy, the evolved-vs-static gap, and pre-evolution distractor selection rate for each model.The distractor-selection rate uses 25% as the chance baseline.
- Experimental conditions: The neutral condition contains 981 items, including 375 static and 606 evolved items.
- Experimental conditions: The short-horizon condition evaluates 0–14-day recall across 120 users and 1,778 items, including 1,545 static and 233 evolved items.
- Experimental conditions: The explicit- and implicit-rewrite conditions each contain 981 items, with 375 static and 606 evolved items evaluated across 25 models.
- Experimental conditions: The low-subtlety condition contains 3,628 items, and relaxing the majority ≥3/5 filter raises overall accuracy while the belief-update pattern persists.