PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Exploration and Persona: Information-Directed Dueling for Test-Time Agent Personalization

Jie Mao, Pingchen Lu, Xiangyi Wang, Hengquan Huang, Shuang Qiu, Min Zhang, Zhongxiang Dai

arXiv:2610.24385v1cs.AIcs.CY

TL;DR

Personalizing LLM responses requires selecting personas for query-dependent preferences from lightweight feedback. ID-TAP combines query-conditioned neural utility estimates, a lightweight BT explorer, and similarity-weighted warm starts to select challengers. Across seven user-conditioned tasks, it achieves the strongest aggregate result among evaluated baselines under a fixed comparison budget.

  • Problem

    Persona usefulness depends on the request, while user preferences may be implicit and noisy, making test-time selection from a persona pool a contextual problem.

    Source: §1 · p. 1

  • Method

    ID-TAP uses an IDS-inspired neural dueling-bandit framework that combines query-conditioned utility gaps, a lightweight BT entropy-reduction proxy, and similarity-weighted warm starts.

    Source: §1 · p. 2

  • Results

    Across seven user-conditioned tasks with automated proxy feedback, ID-TAP achieves the highest aggregate mean among evaluated baselines under a fixed comparison budget.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results support information-directed challenger selection as an effective design for test-time persona adaptation.

    Source: §1 · p. 2

    The method assumes preferences are locally stable over short interaction horizons and adapts within a fixed, pre-constructed persona space and historical context.

    Source: §2.3 · p. 3

Abstract

from arXiv · show

Effective test-time personalization requires LLM agents to adapt to query-dependent user preferences from lightweight interaction. We study persona selection with binary comparisons between candidate responses as the online signal. We present ID-TAP, a scalable, IDS-inspired neural dueling-bandit framework for large discrete persona spaces. ID-TAP combines a query-conditioned neural utility model with a lightweight Bradley-Terry explorer, and transfers comparison evidence across related queries through similarity-weighted warm starts. The neural model shares statistical strength across queries while preserving query-specific scores, and efficient BT updates score hypothetical challengers. Across seven user-conditioned generation benchmarks under reference- and model-based preference simulators, ID-TAP achieves the strongest aggregate performance among the evaluated baselines within a fixed ten-round comparison budget. These results support information-directed challenger selection as an effective design for test-time persona adaptation.

1 INTRODUCTION

The introduction frames persona selection as contextual adaptation from lightweight preference feedback and presents ID-TAP as an information-directed challenger-selection method. Across seven benchmarks, it reports the strongest aggregate result among compared methods under a ten-round budget.

Motivation

  • Persona selection is contextual: a persona useful for brainstorming may be less suitable for editing, while interaction histories offer partial, noisy evidence about user preferences.Semantically similar persona descriptions can still induce meaningfully different responses.Source: §1 · p. 1
  • Preference feedback reduces interaction to choosing between two candidate responses, complementing expressive linguistic feedback for test-time adaptation.The comparison format is also used in RLHF.Source: §1 · p. 1

Related Work

  • In the evaluated persona pools, POHF and PDO’s UCB- and Thompson-sampling-based components revisit challengers more often and achieve lower aggregate performance than ID-TAP’s information-directed rule.Source: §1 · p. 2

Scalable Challenger Acquisition

  • ID-TAP combines query-conditioned neural utility-gap estimates with a lightweight BT entropy-reduction proxy to score challengers efficiently.The separation supports one neural update per observed comparison.Source: §1 · p. 2

Cross-Query Adaptation

  • A similarity-weighted warm start reuses comparison evidence across related requests while preserving query-conditioned utility prediction.Source: §1 · p. 2

Empirical Evaluation

  • Across seven benchmarks and two offline preference simulators, ID-TAP achieves the strongest aggregate result among compared methods under the same ten-round budget.It overtakes the strongest compared baseline after approximately four simulated-feedback rounds on the aggregate generation-task trajectory.Source: §1 · p. 2

2 PROBLEM FORMULATION

The problem is framed as selecting a persona whose utility depends on the current query, using a fixed persona pool built from historical context. Online adaptation uses binary comparisons within a finite interaction budget, under an assumption of locally stable preferences.

2 PROBLEM FORMULATION

  • Persona selection is a contextual, query-level problem with offline construction and online test-time adaptation.Source: §2 · p. 2

2.1 PERSONA AND CONTEXTUAL SPACE

  • A persona summarizes traits, stylistic habits, and task preferences inferred from user history, and candidate utilities vary with each request.The finite persona space is constructed from historical context.Source: §2.1 · p. 2

2.2 OFFLINE–ONLINE FRAMEWORK

  • The offline phase constructs a shared candidate set from historical context, and its candidate strings remain fixed and reusable across subsequent queries.Source: §2.2 · p. 2
  • At test time, the optimal candidate is the persona with highest unknown expected response utility for the current query.Source: §2.2 · p. 2
  • Each new query inherits similarity-weighted BT statistics; the neural model selects a champion and estimates utility gaps while the query-local BT explorer scores hypothetical challengers.Their acquisition ratio selects a challenger; the resulting response pair yields one binary preference and updates both systems.Source: Figure 1 · p. 3
  • The neural score estimates latent utility from observed comparisons.Source: §2.2 · p. 3

2.3 TEST-TIME ADAPTATION WITH PREFERENCE FEEDBACK

  • The method assumes preferences are locally stable over short interaction horizons, keeping the candidate persona space and historical context fixed during a session.Source: §2.3 · p. 3
  • The agent compares a champion and challenger’s responses, receives a binary preference, and aims to return a high-utility persona within budget T.The champion exploits the neural estimate, while the IDS-inspired rule selects the challenger.Source: §2.3 · p. 3

3 METHODOLOGY

ID-TAP uses pairwise user feedback to adapt persona selection at test time, balancing performance-relevant choices with uncertainty reduction. Its design combines neural utility estimation, a lightweight Bradley–Terry explorer, and transfer of comparison evidence across queries.

3.1 ALGORITHM OVERVIEW

  • The agent collects binary comparison feedback over a fixed interaction budget and outputs the highest-scoring persona for the query.Neural parameters persist across a user's queries, while the Bradley–Terry state is initialized and updated locally.Source: §3.1 · p. 4, §3 · p. 4

3.2 IDS-INSPIRED CHALLENGER SELECTION

  • ID-TAP fixes the current greedy persona as champion and selects a challenger by balancing estimated utility gap against a BT-based entropy-reduction proxy.The acquisition ratio favors challengers with a small predicted utility loss and a large estimated reduction in ranking uncertainty.Source: §3.2 · p. 4, §3.1 · p. 4

3.3 DUAL-SYSTEM ARCHITECTURE FOR REAL-TIME EFFICIENCY

  • The neural utility model shares utility estimation across queries, while a query-local Bradley–Terry explorer estimates challenger-side uncertainty from pairwise outcomes.The neural model selects the champion and estimates utility gaps; the BT explorer supplies the entropy-reduction proxy.Source: §3.3 · p. 5
  • For new queries, ID-TAP warm-starts Bradley–Terry counts using similarity-weighted local comparison counts from earlier queries.The inherited count mass is capped at one query-equivalent, while the neural model separately learns from the interaction history.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments compare ID-TAP with personalization and preference-learning baselines under a shared ten-round protocol. They report aggregate performance, feedback-round dynamics, component analyses, and runtime.

4.1 EXPERIMENTAL SETUP

  • Experiments use K = 400 personas per user across LaMP, LongLaMP, PrefEval, and UltraChat, with three independent runs per user.Dataset means, standard deviations, and standard errors are computed across per-user results.Source: §4.1 · p. 6

4.2 BASELINES AND SHARED EVALUATION PROTOCOL

  • Comparisons include PersonaAgent, POHF, and PDO’s D-TS component; bandit methods share personas, response generator, binary feedback, and a ten-round budget.PDO Sampling instantiates the D-TS component on the shared fixed persona pool.Source: §4.2 · p. 7

4.3 PREFERENCE FEEDBACK MECHANISMS

  • Held-out-target and reference-grounded LLM proxies supply binary comparisons; final W.R. measures preference over a naive response from the same backbone.For W.R., response order is randomized and results are reported as the percentage of comparisons favoring the method response.Source: §4.3 · p. 7

4.4 MAIN RESULTS AND ANALYSIS

  • 0.2228 vs 0.2068 cross-dataset raw ROUGE-L averages put ID-TAP ahead of the strongest baseline; it is numerically best in 13 of 14 dataset–metric cells.The exception is LongLaMP-2 W.R., where PersonaAgent is higher.Source: §4.4 · pp. 7–8
  • Figure 2 tracks query-wise and iteration-level ROUGE-L over five LaMP and LongLaMP tasks; ID-TAP passes baselines after about four simulated-feedback rounds.The query-wise average covers LaMP-4, LaMP-5, and LongLaMP-2, -3, and -4.Source: §4.4 · p. 8

4.5 MATCHED COMPONENT ABLATIONS

  • Full ID-TAP exceeds Random Challenger by 0.005, 0.020, and 0.011; both history transfers add on LongLaMP, while PrefEval remains unchanged.Neural history adds 0.010 on each LongLaMP task, and the BT warm start adds a further 0.009 and 0.007.Source: §4.5 · p. 8, Table 2 · p. 8
  • Keyword-conditioned denser persona spaces reduce challenger exploration for POHF and PDO, while ID-TAP changes by at most 1.5 percentage points; ID-TAP is numerically highest across reported model, evaluator, and metric variants.The density analysis covers four datasets.Source: §4.5 · p. 8
  • With K = 100, algorithmic updates take 0.065–0.571 seconds per round versus 0.95–3.15 seconds end-to-end, accounting for approximately 7–22% of latency.The runtime analysis uses 40 sampled users per dataset.Source: §4.5 · p. 9

5 RELATED WORK

Related work spans user-conditioned personalization, feedback-driven test-time adaptation, and exploration for pairwise preferences. ID-TAP applies information-directed acquisition to cross-query persona selection in a fixed pool.

Personalized LLM Agents

  • Personalized LLM methods use retrieval, memory, or prompts to incorporate user context, while ID-TAP selects among fixed persona prompts using pairwise feedback.LaMP standardizes evaluation across personalized language tasks (Salemi et al., 2024).Source: §5 · p. 9

Feedback Mechanisms and Test-Time Adaptation

Exploration in Dueling Bandits

  • Dueling-bandit methods learn from pairwise preferences, and ID-TAP extends information-directed foundations to large persona pools with query-conditioned utility learning and cross-query evidence transfer.POHF uses neural UCB-style acquisition, while PDO combines D-TS with mutation and pruning; the paper’s PDO Sampling baseline uses D-TS in a fixed pool and shared budget.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents ID-TAP as an IDS-inspired approach to persona selection and reports the strongest aggregate mean among evaluated baselines under a fixed comparison budget. It also identifies end-user evaluation and finite-sample analysis as future work.

6 CONCLUSION

  • ID-TAP combines query-conditioned neural utility gaps, a lightweight BT entropy-reduction proxy, and similarity-weighted warm starts; it achieves the highest aggregate mean across seven tasks with automated proxy feedback.The evaluation uses a fixed comparison budget.Source: §6 · p. 9

AI USE STATEMENT

  • Generative AI assisted with formatting, scientific-figure design and implementation, and editorial review, but did not generate or alter reported measurements.The authors state they reviewed all AI-assisted changes and take responsibility for the final text, claims, and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The appendix provides implementation details, prompts, data splits, hyperparameters, analyses, robustness experiments, and timing protocol; public code is available.The conclusion names end-user evaluation and finite-sample analysis as future work.Source: §6 · p. 9, REPRODUCIBILITY STATEMENT · p. 10

A IMPLEMENTATION DETAILS

The appendix details ID-TAP’s cross-query preference bookkeeping and challenger-scoring procedure, alongside its datasets, prompt construction, data splits, and raw ROUGE-L results.

A.1 CROSS-QUERY BT WARM START

  • Cross-query warm starts weight earlier comparison counts by query similarity, with normalization limiting inherited evidence to at most one query-equivalent of mass.An empty history initializes counts to zero and pairwise probabilities to 0.5.Source: §A.1 · p. 13

A.2 MAINTAINED STATISTICS AND SMOOTHED PAIRWISE PROBABILITIES; A.3 BT-MLE STRENGTH ESTIMATION VIA MM ITERATIONS

  • The explorer tracks directed wins and symmetric comparison counts, applies Beta smoothing, then estimates BT strengths with MM iterations while leaving unobserved candidates neutral.Observed-pair probabilities use α = 0.7; strengths are normalized and converted to log scale with numerical safeguards.Source: §A.2 · p. 13, §A.3 · p. 13

A.4 NORMALIZED STRENGTH DISTRIBUTION AND ENTROPY; A.5 OUTCOME SIMULATION AND ENTROPY-REDUCTION PROXY

  • The normalized BT-strength distribution supplies an entropy measure, and the acquisition proxy estimates its reduction by simulating both outcomes of a hypothetical comparison.The proxy clips negative estimated reductions to zero.Source: §A.4 · p. 14, §A.5 · p. 14

A.6 REPRODUCIBLE PROCEDURE (PSEUDO-CODE)

  • For each candidate challenger, the reproducible procedure performs two hypothetical BT re-estimations, requiring 2(K − 1) re-estimations per round with a fixed number of MM iterations.The pseudocode evaluates each simulated outcome and returns the clipped difference between current and expected post-update entropy.Source: §A.6 · p. 14

B.1 DATASET

  • Evaluation spans LaMP and LongLaMP generation tasks, PrefEval dialogue response generation, and UltraChat next-query prediction.LaMP includes personalized headline and scholarly-title generation; LongLaMP includes abstract, product-review, and topic generation.Source: §B.1 · p. 15

B.2 PROMPT TEMPLATES

  • The prompt templates build persona candidates through bounded rephrasing and condition responses on task details, persona summaries, history keywords, and interaction history.The rephrasing procedure resets to the original context every 15 iterations to limit semantic drift.Source: §B.2 · p. 16

B.3 DATA SPLITTING FOR LAMP DATASETS

  • For LaMP-based datasets, profile entries are split into history context and held-out query/output pairs, whose references support proxy-user comparisons and evaluation.The split uses profile-size thresholds and caps the number of extracted input/output pairs.Source: §B.3 · p. 17

B.4 UNNORMALIZED ROUGE-L STATISTICS

  • ID-TAP has the highest raw ROUGE-L mean on all seven datasets, averaging 0.2228 versus 0.2068 for the strongest baseline.These unnormalized values complement the normalized main results.Source: §B.4 · p. 18

B.5 PERSONA SPACE QUALITY ANALYSIS

The persona pools are moderately dense, contain few near-duplicates, and show weak cluster separation; the appendix also records ID-TAP’s model and training settings.

B.5 PERSONA SPACE QUALITY ANALYSIS

  • Across datasets, mean pairwise cosine similarity ranges from 0.657 to 0.716, while near-duplicate rates remain below 0.20% and effective-arm ratios range from 94.4% to 98.2%.The reported similarity intervals span roughly 0.22 between the 10th and 90th percentiles.Source: §B.5 · p. 18
  • K-means silhouette scores at K=20 range from 0.097 to 0.125, indicating weak separation into discrete clusters across the persona pools.Together with the similarity statistics, the analysis characterizes the pools as dense and weakly clustered.Source: §B.5 · p. 18

B.6 HYPERPARAMETERS

  • The utility model is a single-hidden-layer feedforward network over concatenated query and persona embeddings, with dropout rate 0.05 and GELU activation.The embedding dimension is 1024 in standard mode and 2048 in contextual mode.Source: §B.6 · pp. 18–19
  • Training uses AdamW with learning rate 5 × 10^-4, while algorithm settings include BT smoothing α = 0.7, temperature τ = 4.5, and similarity-transfer parameters βsim = 6 and μsim = 0.8.The configuration also specifies LLM settings for generation and preference evaluation.Source: §B.6 · p. 19, Table 6 · p. 20

C.1 CONSTRUCTION AND EXPLORATION DIAGNOSTIC

The appendix tests construction and exploration patterns, their association with performance margin, and whether ID-TAP’s result extends to smaller persona pools and an alternative construction.

C.1 CONSTRUCTION AND EXPLORATION DIAGNOSTIC

  • Across four datasets, keyword-conditioned construction increases mean persona similarity and accompanies lower conventional exploration, while ID-TAP’s exploration rate changes by at most 1.5 percentage points.The within-dataset construction comparison associates denser persona spaces with reduced conventional exploration, alongside comparatively stable ID-TAP exploration.Source: §C.1 · pp. 20–21, Table 7 · p. 20, Table 8 · p. 21

C.2 EXPLORATION–PERFORMANCE ASSOCIATION

  • Datasets with lower POHF exploration under keyword-conditioned construction also show a larger reported ID-TAP margin, a descriptive association rather than a causal result.Table 10 summarizes the association between conventional exploration and ID-TAP’s ROUGE-L margin over the strongest baseline.Source: §C.2 · p. 21, Table 10 · p. 21

C.3 SENSITIVITY TO PERSONA-SPACE SIZE AND CONSTRUCTION

  • At K = 100, ID-TAP achieves the highest mean ROUGE-L under both iterative-rephrasing and keyword-conditioned persona construction.These experiments use 40 sampled users per dataset, compared with K = 400 in the main experiments.Source: §C.3 · pp. 21–22, Table 11 · p. 21, Table 12 · p. 22

D ROBUSTNESS TO PREFERENCE AND EVALUATION MODELS

ID-TAP’s reported ranking persists across preference-generation and evaluation configurations, including an evaluation metric independent of the optimization signal.

D ROBUSTNESS TO PREFERENCE AND EVALUATION MODELS

E RUNTIME AND TIMING PROTOCOL

The appendix reports the runtime measurement protocol and timing, illustrates persona-bank examples, and points to dataset-level performance and trajectory views.

E RUNTIME AND TIMING PROTOCOL

F.1 LAMP-4

  • Three LaMP-4 examples portray a user tracking campaign tactics and candidate clashes, while Figure 3 illustrates diversity among same-domain persona summaries.The example emphasizes concrete developments, strategic competition, and headlines framing events as consequential clashes.Source: §F.1 · p. 23, Figure 3 · p. 24

F.2 LONGLAMP-4

  • LongLaMP-4 examples depict a thinker prioritizing functional hierarchy and practical utility across game design, technology, physics, and finance.The persona examples favor tangible benchmarks and grounded explanations over sentimental or abstract reasoning.Source: §F.2 · p. 24, Figure 4 · p. 25

G MORE EXPERIMENTAL RESULTS

  • Figures 5 and 6 provide individual-dataset results and dataset-specific trajectories beyond the main paper’s aggregate view.Figure 5 reports individual-dataset results; Figure 6 shows how performance evolves within the fixed comparison budget.Source: G MORE EXPERIMENTAL RESULTS · p. 26

G.1 DETAILED PERFORMANCE COMPARISON ACROSS DATASETS

The appendix organizes detailed comparisons across benchmark categories and provides both query-level performance and iteration-level trajectories for the evaluated benchmarks.

G.1 DETAILED PERFORMANCE COMPARISON ACROSS DATASETS

  • Figure 5 compares three categories: LaMP generation, LongLaMP generation, and multi-turn dialogue benchmarks.The listed tasks cover personalized headline and scholarly-title generation; abstract, review, and topic generation; and UltraChat and PrefEval.Source: §G.1 · p. 26, Figure 5 · p. 26
  • Figure 5 presents query-level performance across all evaluated benchmarks.It underlies the aggregate view in the main paper.Source: Figure 5 · p. 26

G.2 DETAILED ITERATION-LEVEL TRAJECTORIES

  • Figure 6 presents iteration-level trajectories for each benchmark under automated proxy feedback.The trajectories show performance evolving within the fixed comparison budget and provide dataset-specific detail.Source: §G.2 · p. 26, Figure 6 · p. 27
Loading 2610.24385v1…