PaperlayerView exact revision on arXiv

Source-linked AI summary

Bounded Personas Match Retrieval on Classification but Not Regression for a Frozen Agent

JaeHa Yoon, Minjun Park, Seoyeon Kim, Jiwoo Lee, Hyunwoo Choi, Dohyun Kang

arXiv:2609.02890v1cs.CL

TL;DR

Personalized agents must convert labeled user histories into behavior on new requests, but it is unclear when bounded distillation can match retrieval. PersonaLink recursively refines a bounded three-field persona with a frozen agent and held-out self-evaluation. It ties retrieval on LaMP-2 classification but loses on LaMP-3 regression, while recursion saturates after one pass and the paper’s scope remains limited.

  • Problem

    Whether a bounded, query-independent distilled persona can match retrieval across task types has not been cleanly characterized.

  • Method

    PersonaLink recursively distills each user’s labeled history into a bounded three-field persona, rewriting it from held-out errors under a keep-better gate with one frozen backbone.

  • Results

    PersonaLink ties retrieval on LaMP-2 classification but is decisively beaten on LaMP-3 regression, while retrieval continues improving as k grows.

  • Takeaways & Limitations

    Bounded distillation is sufficient for label classification but not fine-grained regression or scaling with additional history.

  • Takeaways & Limitations

    The evidence covers two English LaMP tasks and one 7B-class frozen backbone, so the asymmetry may shift with scale or task family.

Abstract

from arXiv · show

A personalized language agent must convert a user's interaction history into behavior on each new request at inference time. Two strategies dominate. Retrieval pulls a few of the user's most relevant past items into the prompt, which is accurate but pays a per-query selection and context cost that grows with the history. Distillation instead compresses the history once into a compact natural-language persona, which is bounded, query-independent, and interpretable, but is widely assumed to sacrifice accuracy. Whether, and on which tasks, a distilled persona can match retrieval has not been characterized cleanly. We introduce PersonaLink, a training-free method that distills a user's history into a bounded three-field persona and recursively refines it: each pass self-evaluates the frozen agent on a held-out slice of the user's own labeled history, rewrites the persona from its errors, and keeps the result only when it does not regress on that slice. Because every comparison shares one frozen 7B backbone and differs only in what is placed in context, the design isolates the effect of representation from that of the model. The result is a clear task-type asymmetry. On 200 users of LaMP-2 (15-way news categorization), PersonaLink reaches 0.745-0.755 accuracy, statistically indistinguishable from BM25 retrieval (0.760-0.765).

1 Introduction

The paper asks when a bounded distilled persona can match retrieval and introduces PersonaLink to test that question under a frozen backbone. Results show a task-type asymmetry: distillation matches retrieval for classification but not regression, while recursive refinement saturates quickly.

  • Motivation: Retrieval offers strong personalization but incurs per-query selection and context costs, whereas distillation is bounded, query-independent, and interpretable.The paper studies whether distillation can match retrieval despite its assumed accuracy disadvantage.
  • Method: PersonaLink recursively rewrites a bounded three-field persona from errors on held-out labeled history, accepting updates only when they do not regress.All methods use one frozen backbone and differ only in the context text, isolating representation effects.
  • Results: 0.745–0.755 accuracy on LaMP-2 classification was statistically indistinguishable from BM25 RAG at k=3, 5 (0.760/0.765).This is the paper’s principal classification comparison.
  • Results: 0.455 MAE versus RAG-5’s 0.285 on LaMP-3 regression, while retrieval reached 0.250 MAE at k=20 with no plateau.The paper reports retrieval as the stronger method overall and PersonaLink as a characterization rather than a state-of-the-art method.
  • Recursion: Recursion adds little beyond the first pass as persona updates contract toward a per-user fixed point.The measured contraction has modulus L=0.144 and R2=0.979.
  • Conclusion: A bounded, query-independent persona suffices for selecting labels but not for estimating fine-grained quantities.The paper explicitly frames this as its characterization of the distill-versus-retrieve regime.

2 Preliminary: the personalized prediction setting

Personalized prediction maps a user’s labeled history to behavior on a new unlabeled request. The paper compares no personalization, retrieval, distillation, controls, and an oracle across classification and ordinal regression tasks.

  • Setting: A personalized agent must infer the user’s behavior from past labeled items because the new item’s label is unavailable.The same input can receive different labels or ratings for different users.
  • Methods: Retrieval selects the k most relevant history items as demonstrations, so its injected context depends on the query and grows with k.BM25 is used for retrieval in the personalized prediction setting.
  • Methods: Distillation compresses the user history once into a persona whose injected context is fixed per user and independent of the query.The random-user control keeps token budget fixed, while the oracle supplies the full true profile as a ceiling.
  • Evaluation: LaMP-2 is 15-way news classification scored by Accuracy and macro-F1, while LaMP-3 is ordinal 1–5 regression scored by MAE and RMSE.For LaMP-3, lower MAE and RMSE are better.

3 PERSONALINK: a bounded, query-independent persona

PERSONALINK represents each user with a compact three-field PERSONASTATE whose size remains bounded regardless of history length. It constructs this persona through training-free, label-grounded self-evaluation of a frozen agent.

  • PERSONASTATE: PERSONASTATE contains a preference summary, representative input→label exemplars, and conditional decision rules.Each field is natural-language text describing preferences, examples, or heuristics induced from past items.
  • PERSONASTATE: Hard length caps keep the rendered persona bounded regardless of the user’s history length.This contrasts with add-only memories and retrieval contexts whose footprints grow with experience or k.
  • Construction: The method constructs the persona with a training-free loop over the user’s own profile, using known labels for deterministic verification.Figure 2 depicts the persona as a preamble before the frozen agent’s evaluation and rewriting steps.
  • Construction: Each refinement pass renders the current PERSONASTATE as a single preamble and evaluates the frozen agent on a held-out slice of labeled history.The resulting error set identifies profile items whose predictions disagree with their known labels.

3. Rewrite. Feed Eu together with p(r)

After self-evaluation, PersonaLink rewrites the persona using its errors and retains the candidate only when held-out performance does not regress. Refinement depth separates the initial distillation pass from recursive self-correction.

  • Rewrite: The rewrite step feeds the error set together with the current persona to the agent, producing a candidate persona within the field length caps.Fields may be edited or dropped to address the identified errors.
  • Keep-better gate: The keep-better gate accepts a candidate only when its held-out score is no worse than the incumbent.Otherwise, the current persona is retained under a strict held-out best-update rule.
  • Refinement depth: Depth r=0 is the empty no-persona control, r=1 is a single distillation pass, and r≥2 denotes recursive error-driven self-correction.After r passes, the kept-best persona answers the actual test items.
  • Refinement depth: The test item is excluded from the held-out slice, and only the persona text changes across refinement depths.This design isolates the effect of refinement from the identical test question and frozen agent.

4 Learning to personalize by recursion

The recursion uses held-out user history to refine a persona under a strict keep-better gate, guaranteeing non-decreasing held-out scores while leaving unseen-test transfer empirical. In this evaluation, the first pass is compute-optimal because later refinement adds no significant accuracy and the measured contraction supports saturation rather than an unconditional theorem.

  • Optimization and analysis: The refinement loop optimizes persona text using the user’s own held-out signal.The method models each pass as a map on an embedding of the persona text and evaluates test-set behavior empirically.
  • Monotone safety: The strict keep-better gate makes the held-out score non-decreasing across refinement passes.If no candidate improves over the empty persona, the procedure remains equivalent to no personalization.
  • Scope: The proposition guarantees held-out selection behavior, not necessarily accuracy on unseen test items.The paper explicitly treats transfer to test items as an empirical question rather than a consequence of the gate.
  • Evaluation setup: The evaluation holds the backbone, decoding, users, splits, and scoring fixed while comparing classification and regression regimes.The protocol uses one frozen 7B-class backbone and identical greedy decoding; implementation includes robust label parsing without a generative judge.
  • Convergence account: Under stated assumptions A1 and A2, the refinement iterates converge geometrically and per-pass changes decay like L^r.The paper presents this as a falsifiable Banach-style account and measures the relevant contraction quantity rather than asserting it unconditionally.
  • Compute-optimal depth: The first pass is compute-optimal here because later passes add no significant accuracy.The empirical result is r=1, while r=2 and r=3 add no significant accuracy under constant per-pass cost.

5 Experiments

Experiments compare bounded persona distillation with retrieval under one frozen backbone across classification and regression. The persona matches retrieval on classification at constant cost, but retrieval dominates regression and continues improving with added context; refinement gains plateau after one pass.

  • Setup: All methods share one frozen 7B-class backbone, identical greedy decoding, deterministic metrics, and n=200 users per task.Significance uses McNemar’s test for LaMP-2 and paired bootstrap for LaMP-3.
  • Classification: 0.745–0.755 LaMP-2 accuracy for PERSONALINK is statistically indistinguishable from BM25 RAG-3 and RAG-5 at 0.760 and 0.765.The comparison concerns 15-way label prediction and uses paired tests.
  • Regression: 0.455 LaMP-3 MAE for PERSONALINK versus 0.285 for RAG-5 shows retrieval’s significant advantage on ordinal regression.The gap is ∆MAE=0.17 with p<10−4; RAG-3 reaches 0.290 MAE.
  • Cost and scaling: 0.045 additional LaMP-2 accuracy from RAG-20 costs roughly 8× the per-query context, while the persona remains fixed at approximately 250 tokens.The persona is built once, so its injection cost is O(1)/query.
  • Recursion: +0.10 LaMP-2 accuracy occurs from r=0 to r=1, while r≥1 remains flat within the shaded intervals.The first pass captures the personalization gain; later refinement does not add visible test-set improvement.

6 Analysis

The analysis links refinement saturation to contraction toward user-specific fixed points and examines why task type determines whether bounded personas suffice. Qualitative edits remain compact and interpretable, but retrieval remains the strongest non-oracle method.

  • Recursive convergence: 0.90→0.21→0.02 mean embedding displacement and L=0.144 with R2=0.979 indicate geometric contraction toward per-user fixed points.The authors present this as measured contraction rather than a proven theorem.
  • Task-type asymmetry: Classification needs a sorting rule, whereas regression requires a calibrated numeric scale that bounded prose cannot encode with rating-level granularity.Concrete retrieved examples therefore provide information that the bounded persona loses on regression.
  • Comparison: Strong BM25 RAG is the best non-oracle method in both tasks, while recursive refinement does not surpass it.The broader comparison uses 11 methods with one frozen backbone and identical greedy decoding.
  • Field ablation: Removing decision rules hurts LaMP-2 most, while removing exemplars hurts LaMP-3 most, matching the distinct information demands of the tasks.All ablations retain the approximately 250-token cap and remain above the NOPERS floor.
  • Qualitative personas: By r=2, persona rewrites are near-paraphrases of r=1, consistent with the measured contraction and depth-scaling plateau.The qualitative edits include promoting an under-used category and adding a rule for defective-but-usable reviews.

7 Related work

Related work frames PERSONALINK against retrieval, textual and parameter-efficient distillation, recursive self-refinement, memory systems, and personalized alignment. The paper differs by keeping the backbone frozen and adapting only a bounded textual persona.

  • Personalizing LLMs: LaMP established personalized prediction from user profiles, with retrieval as a strong baseline later refined by retrieval-optimization methods.The paper characterizes when bounded personas match retrieval rather than proposing a stronger personalizer.
  • Distillation: Prior distillation methods summarize profiles, personalize parameters, learn soft user embeddings, store long-term memory, or infer user cues at inference.PERSONALINK instead uses a deliberately bounded textual representation.
  • Role-playing personas: The paper targets user personas for personalization rather than assigned-character role-play settings.Role-playing persona agents constitute a related but distinct line of work.
  • Personalized alignment: Personalized alignment methods adapt model weights or rewards, while this paper keeps the backbone frozen and adapts only a textual persona.This distinguishes the method’s representation-level intervention from parameter or reward adaptation.
  • Recursive agents: Recursive refinement builds on self-correction, prompt evolution, self-training, held-out self-evolution, and recursion as a way to spend more test-time compute.The paper applies these ideas to iterative persona refinement.
  • Agent memory: Add-only context memories accumulate growing text, whereas PERSONALINK deliberately uses a bounded, query-independent footprint.The reported trade-off is that boundedness costs the regression task and the scaling retrieval enjoys.

8 Conclusion and limitations

PERSONALINK provides a bounded, interpretable, query-independent persona that ties retrieval on LaMP-2 classification but is beaten on LaMP-3 regression. Its scope is limited by retrieval’s overall advantage, the single frozen 7B-class backbone, held-out-only selection, and evaluation on two English LaMP tasks.

  • Conclusion: PERSONALINK ties retrieval on LaMP-2 classification but is decisively beaten on LaMP-3 regression.The paper characterizes when bounded distillation is sufficient rather than presenting it as a new state of the art.
  • Conclusion: Bounded distillation is sufficient for label classification, but not for fine-grained regression or at scale.The proposed artifact is cheap, inspectable, bounded, and query-independent.
  • Limitations: Retrieval wins overall and remains the best non-oracle method throughout the reported results.
  • Limitations: The evaluation uses one frozen 7B-class backbone, two English LaMP tasks, and held-out-only gating, limiting generality and test-set guarantees.

A Proof of Proposition 1

The proof shows that the keep-better gate makes the held-out score of the kept persona non-decreasing across refinement passes. The accompanying measurement estimates contraction from persona displacement while explicitly treating it as empirical rather than a global theorem.

  • Proof of Proposition 1: A_r ≥ A_{r−1} for every r ≥ 1 because the gate keeps a candidate only when its held-out score is at least the incumbent’s.The induction starts from the empty persona and covers both candidate acceptance and incumbent retention.
  • Contraction measurement: The experiments embed each kept persona, compute per-pass displacement, average across 15 LaMP-2 users, and regress log displacement on step index.
  • Contraction measurement: 0.90→0.21→0.02 mean displacement and L=0.144 with R2=0.979 quantify rapid contraction across passes.The fit estimates the contraction modulus under assumptions A1–A2.
  • Experimental setup: All comparisons use one frozen 7B-class backbone with greedy decoding, differing only in the in-context text.Methods include retrieved demonstrations, a persona, or nothing; LaMP-2 uses Accuracy and macro-F1, while LaMP-3 uses MAE and RMSE.

D Qualitative evolved personas

The qualitative traces show short, legible, bounded personas for both classification and regression users. The classification example stabilizes after one pass, while the regression example accepts later rewrites without matching retrieval’s concrete ratings.

  • Qualitative evolved personas: The qualitative appendix presents two verbatim kept PERSONASTATEs taken from logged per-user refinement traces.
  • LaMP-2 classification: For LaMP-2 user 1136, the first pass raises held-out accuracy from 0.875 to 1.000, and r=1, 2, 3 personas are byte-identical.
  • LaMP-2 classification: The classification persona encodes politics-focused preferences and explicit rules for mapping named topics and policy issues to politics.
  • LaMP-3 regression: For LaMP-3 user 211070, two r=2 rewrites are accepted and held-out accuracy rises from 0.50 to 0.75.The resulting persona encodes a scoring scale, but bounded prose rules cannot match retrieved concrete ratings on MAE.
  • LaMP-3 regression: The regression persona maps qualitative review signals to scores from 3.5 to 5, including writing quality, emotional impact, plot engagement, and depth.
  • Qualitative evolved personas: Both personas remain short, legible, and bounded, while the LaMP-2 trace makes saturation textually visible through identical later personas.
Loading 2609.02890v1…