PaperlayerView exact revision on arXiv

Source-linked AI summary

Persona Learning from Feedback for Personalized AI Assistants

Yiran Shen, Yu Xia, Liuyi Yao, Prithviraj Ammanabrolu

arXiv:2610.38561v1cs.LGcs.AIcs.CL

TL;DR

Personalization must adapt how an assistant helps to users’ preferences and task contexts, but persona descriptions alone do not specify response-specific corrections or which behaviors fit a request. PLF learns reusable personas from feedback and routes requests or dialogue to their adapters; across single-turn personalization and multi-turn tutoring, it improves model-judged preference fit over baselines.

  • Problem

    Persona descriptions alone do not specify which parts of a response to revise or preserve, while the appropriate correction depends on the request’s preferences.

    Source: §1 · pp. 1–2

  • Method

    PLF learns independent persona adapters through gated, feedback-conditioned self-distillation and uses a supervised router to select or compose adapters from request or dialogue context.

    Source: §1 · p. 2

  • Results

    Across single-turn personalization and multi-turn tutoring, PLF improves model-judged preference fit over fixed-persona policies and other learning baselines.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The results support separating persona learning from contextual selection to personalize assistance without user-specific parameter updates.

    Source: §5 · p. 10

    Tutoring uses simulated students, so their satisfaction measures are not evidence of human learning outcomes.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Personalizing AI assistants requires adapting their behavior to users' preferences and task contexts. Existing work elicits assistant personas through persona prompting or finetuning on persona-conditioned generations. However, persona descriptions alone do not identify what to change or preserve in a particular response, or which behaviors suit the current request. We introduce $\textit{Persona Learning from Feedback (PLF)}$, a framework that learns reusable, preference-grounded assistant personas as independent LoRA adapters on a shared frozen backbone. During training, a frozen self-teacher conditions on persona specifications and response-specific feedback to provide token-level supervision on each adapter's own generated prefixes. A feedback gate focuses corrective updates on responses judged to need improvement. At deployment, a supervised router uses the request or dialogue to select a single adapter or a fixed composition, without user-specific finetuning. We evaluate PLF on single-turn personalization with model-generated critiques and multi-turn tutoring with simulated students' reactions. Across both settings, PLF improves model-judged preference fit over fixed-persona policies and strong baselines including persona prompting and persona distillation, demonstrating how feedback-trained behaviors can be reused for adaptive assistant personalization without degrading general capabilities.

1 INTRODUCTION

The introduction frames personalization as adapting how an assistant helps to the preferences and task requirements in each request or dialogue. It presents PLF as learning reusable personas from response-specific feedback and selecting them from context.

1 INTRODUCTION

  • Persona descriptions specify desired behaviors but do not identify which parts of a particular response to revise or preserve.The introduction illustrates that different drafts can require different corrections despite sharing a persona description.Source: §1 · p. 1
  • Because the preferred correction depends on the request, PLF addresses both learning reusable personas from response-specific feedback and selecting personas from available context.For example, shortening may help one request but make another less useful.Source: §1 · p. 2
  • PLF trains independent LoRA persona adapters with a frozen self-teacher, gates updates to responses needing improvement, and uses a supervised router to select or compose adapters.The adapters are intended to express their personas without an additional persona prompt at inference or per-user finetuning.Source: §1 · p. 2, Figure 1 · p. 2
  • Across single-turn personalization and multi-turn tutoring, PLF improves model-judged preference fit over strong baselines, while selected adapter pairs can improve fit and be reused as preferences change.The evaluation uses model-generated critiques in the single-turn setting and simulated students’ replies in tutoring.Source: §1 · pp. 2–3

2 RELATED WORK

Related work covers persona elicitation and user personalization, learning from feedback, and modular adaptation. PLF is positioned at the intersection of response-specific feedback and contextual persona selection.

2 RELATED WORK

  • Persona prompting and persona-conditioned training elicit behavioral tendencies, while user-history and feedback methods personalize responses; these approaches distinguish eliciting behavior from deciding when to use it.The cited approaches include role or style prompting (Xu et al., 2023), persona training (Hu et al., 2026), and user-information methods (Spohn et al., 2026).Source: §2 · p. 3
  • Feedback learning uses critiques for revision or training, and on-policy distillation supervises student-generated trajectories, including with privileged textual feedback.Related personalization work distills subsequent user messages or uses natural-language feedback for personalized question answering (Buening et al., 2026).Source: §2 · p. 3
  • Modular personalization combines or routes parameter-efficient adaptations, but prior work leaves open when modularization is preferable to a shared conditional policy.Examples include preference-dimension model merging (Jang et al., 2023), LoRA module selection or combination, and group- or cluster-level personalized adaptations (Song et al., 2026b).Source: §2 · p. 3

3 METHOD

PLF separates learning reusable persona behavior from selecting personas for each request or interaction. It learns adapters from gated, response-specific feedback and routes observable context to a single adapter or a fixed composition.

3 METHOD

  • PLF represents each reusable assistance behavior with an independent LoRA persona head on a shared frozen base model, keeping its updates separately selectable and composable.The student does not receive persona descriptions in its prompt; those descriptions instead guide feedback and teacher supervision.Source: §3 · pp. 3–4

3.1 RESPONSE-SPECIFIC FEEDBACK

  • Response-specific feedback comes from critiques in single-turn tasks or simulated users’ conversational reactions, with a gate excluding accepted responses and unusable critiques from updates.The feedback procedure sets the gate to zero when no further correction is needed and one otherwise.Source: §3.1 · p. 4

3.2 LEARNING PERSONAS FROM FEEDBACK

  • A frozen base-model self-teacher adds persona specifications and feedback as hindsight guidance, then provides token-level targets on student-generated prefixes for gated distillation.The feedback informs next-token targets without requiring a separately generated revised answer; examples with zero gates contribute no distillation loss.Source: §3.2 · p. 4

3.3 PREFERENCE-BASED ROUTING AND DEPLOYMENT

  • A supervised router predicts personas from observable request or dialogue representations, selects at most one alternative per preference axis, and uses fixed coefficients to compose selected adapters.Composition averages axis-specific updates built from the midpoint of opposing adapters and a shift toward the selected persona; coefficients are tuned on validation data.Source: §3.3 · p. 5

4 EXPERIMENTS

PLF is evaluated in single-turn personalization and multi-turn tutoring, where it learns persona heads from response-specific feedback and routes them using context. Across both settings, it improves preference fit over fixed-persona policies and learning or prompting baselines, with analyses of combined and changing preferences, training behavior, and general capability retention.

4.1 EXPERIMENTAL SETUP

  • Single-turn requests use warm–impersonal and concise–thorough preferences, while tutoring uses direct instruction and interactive guidance; each setting trains and routes persona heads.The single-turn critic provides response-specific feedback, whereas the simulated student’s next turn supplies tutoring feedback.Source: §4.1 · pp. 6–7

4.2 CAN NATURAL LANGUAGE FEEDBACK LEARN USEFUL PERSONAS?

  • 61.7% win against persona prompting in single-turn requests and 56.2% against persona distillation in tutoring; Figure 2 covers both settings.The tutoring setting uses student reactions rather than explicit critiques as feedback.Source: Figure 2 · p. 7, §4.2 · p. 7

4.3 CAN LEARNED PERSONAS SERVE COMBINED OR CHANGING PREFERENCES?

  • In single-turn composition, warm-concise shows the clearest improvement over both constituent heads by combining brevity with encouragement, and its advantage persists when routing errors are included.On validation, calibrating voice and depth weights outperforms simple summation or averaging, with weaker voice weights selected for thorough pairs.Source: Table 1 · p. 8, §4.3 · p. 8
  • 76.7% whole-lesson satisfaction versus 79.2% for the oracle; PLF beats fixed heads and the base in both switch directions and rebounds at the first turn it can use new-preference feedback.The router was trained only on fixed-preference dialogues, and the switch is unannounced.Source: Figure 3 · p. 8, §4.3 · p. 8

4.4 ABLATIONS: FEEDBACK GATING AND KL DIRECTION

  • Warm and impersonal adapters reach high critic approval early, while concise improves more gradually and thorough ends lowest; tutoring satisfaction rises rapidly before plateauing.These are training-feedback trends, not held-out performance; tutoring satisfaction is measured before the final gate’s additional checks.Source: Figure 4 · p. 9, §4.4 · p. 9

4.5 GENERAL CAPABILITY RETENTION

  • PLF performs comparably to the base model on all four general benchmarks in both settings, with no statistically significant degradation.Unlike feedback imitation, PLF has no significant MMLU-Pro drop in single-turn personalization and records the highest observed MT-Bench score in tutoring.Source: Table 4 · p. 10, §4.5 · p. 10

4.4 ABLATIONS: FEEDBACK GATING AND KL DIRECTION

  • Feedback gating yields a 61% win rate over ungated single-turn training and a 55% win rate for interactive tutoring, despite similar critic approval rates.The preference advantage persists on held-out evaluations as earlier training gains diminish; the direct persona remains near parity.Source: §4.4 · p. 10
  • 55% preference win rate for forward KL over reverse KL on 499 routed requests, with the strongest advantages for impersonal-only and impersonal–thorough requests.Warm and concise results remain inconclusive; reverse-KL thorough responses are longer without a preference advantage.Source: §4.4 · p. 10

5 CONCLUSION

PLF learns reusable persona adapters from response-specific feedback and selects or combines them using contextual preference evidence. Across single-turn personalization and multi-turn tutoring, it improves preference fit, while the authors note limits on what the tutoring results establish.

5 CONCLUSION

  • PLF improves model-judged preference fit over fixed persona policies, the base model, persona prompting, and other learning baselines across single-turn personalization and multi-turn tutoring.The learned personas also support combined and changing preferences, without user-specific parameter updates.Source: §5 · p. 10
  • The authors used generative AI tools for language editing and presentation, not for research ideas, analyses, result interpretation, or scientific conclusions.They state that authors reviewed and edited all AI-assisted text and take responsibility for the final content.Source: AI USE STATEMENT · p. 11
  • Simulated-student satisfaction should not be interpreted as evidence of human learning outcomes, and PLF is not intended for high-stakes profiling or decision making.The work also notes that systems may incorrectly infer preferences or reduce them to coarse categories.Source: ETHICS STATEMENT · p. 11
  • The paper provides algorithms, configurations, data-processing and split procedures, routing and decoding settings, evaluation protocols, prompts, and statistical procedures for reproducing its experiments.It also reports model versions, random seeds, hardware settings, and dataset revisions where applicable.Source: REPRODUCIBILITY STATEMENT · p. 11

A PLF IMPLEMENTATION DETAILS

The implementation uses truncated frozen-teacher targets, independent LoRA persona adapters, and signed weight-space composition. The section also specifies training and evaluation configurations for the experiments.

A.1 TRUNCATED TEACHER TARGETS

  • Teacher targets retain the top 256 tokens, while students use full-vocabulary normalization and targets remain fixed during each update.Valid tokens count in the loss even when the feedback gate is zero, and the loss is not renormalized over only gate-selected examples.Source: §A.1 · p. 15

A.2 LORA PARAMETERIZATION AND EXACT COMPOSITION

  • Persona adapters have disjoint parameters on a frozen shared backbone; zero-initialized B matrices make each persona policy initially recover the base model.The paired Qwen3-4B implementation concatenates four rank-32 adapters into rank 128 at scale 256, preserving the scale-to-rank ratio of 2.Source: §A.2 · p. 15

A.3 SIGNED PAIR COMPOSITION

  • Signed pair composition averages four participating updates and adds one weighted contrast per selected preference axis; coefficients sum to one but may be negative.The negative coefficient is possible when the corresponding contrast weight exceeds one-half.Source: §A.3 · p. 15

B ADDITIONAL EXPERIMENT DETAILS

  • Both settings use rank-32 LoRA adapters at scale 64, no adapter dropout, and adapters on all attention and MLP projections.Heads sample at temperature 0.7 and train with AdamW at learning rate 10^-5 and 3% warmup; the frozen backbone supplies top-256-token teacher targets.Source: B ADDITIONAL EXPERIMENT DETAILS · p. 15

B.1 SINGLE-TURN DATA, TRAINING, AND ROUTING

This section specifies the single-turn and tutoring data, feedback-based head training, routing, and evaluation protocols. It also describes the prompts, preference conditions and judges, and procedures for reporting supplementary preference results.

B.1 SINGLE-TURN DATA, TRAINING, AND ROUTING

  • The single-turn corpus starts from PRISM opening requests and screened response-style preferences, which define fixed warm, impersonal, concise, and thorough persona cards.The final head-training corpus contains 6,400 requests from 1,242 users; the cards use training-user preferences and are not supplied to the student or router.Source: §B.1 · p. 16, Table 5 · p. 16
  • Single-turn PLF trains four independent LoRA heads on a frozen Qwen3-4B backbone, using feedback-conditioned teacher token targets and a gate that gives zero loss when changes are not needed.The critic judges persona fit and request fulfillment, not factual correctness; valid tokens remain in the denominator when a response is gated out. The step-60 heads are used for reported evaluations.Source: §B.1 · p. 17
  • A supervised single-turn router predicts persona indicators from the rewritten request and selects one head or a fixed two-head composition; it matched assigned personas on 426 of 499 test requests.At inference, the router and heads receive only the rewritten request, without a condition label, survey preference, or persona card.Source: §B.1 · pp. 17–18

B.2 MULTI-TURN TUTORING DATA, TRAINING, AND ROUTING

  • Tutoring uses 4,524 training, 200 validation, and 500 test dialogues, with hidden direct-instruction or interactive-guidance preferences and fixed simulated-student reaction styles.Only the opening teacher turn comes from the dataset; subsequent student and tutor turns are generated, and original preference and reaction annotations are unused.Source: §B.2 · p. 18
  • Tutoring heads train for 100 rollout cycles, with student replies serving as feedback for the preceding tutor turn and the teacher responding again to the original student message.Each head tutors 32 openings for eight turns per cycle; the final heads are evaluated.Source: §B.2 · p. 18
  • The tutoring router predicts preference from tutor-context features and always selects one of two heads; human annotators also calibrate the frozen student-simulator prompt on held-out contexts.Router training crosses generating heads with student types to prevent the generating head from determining the target label, and uses no validation or test topics for fitting or epoch selection.Source: §B.2 · pp. 18–19

B.3 EVALUATION AND STATISTICAL REPORTING

  • Evaluation measures pairwise preference win rate across both presentation orders, with ties for order disagreement, explicit ties, identical single-turn responses, and conflicting switching verdicts.The section also defines tutoring satisfaction as replies not flagged as objections and factual-error rate as tutor turns flagged by a fact-checking judge; reported intervals use bootstrap resampling.Source: §B.3 · p. 19

C.1 PERSONA DESCRIPTIONS; C.2 STUDENT, CRITIC, AND DISTILLATION TEACHER; C.3 BASELINE PROMPTS; C.4 PREFERENCE-EMBEDDED REQUEST CONSTRUCTION; C.5 PAIRWISE PREFERENCE JUDGE

  • The prompt appendices specify four one-axis persona cards, critic feedback and teacher-scoring contexts, and baseline distinctions: persona distillation scores without feedback, while feedback imitation trains on teacher-generated responses.Preference-embedded requests minimally express assigned conditions; the router and response policy see only the rewritten request, while the judge receives the assigned preference text and both responses.Source: §C.1 · p. 20, §C.2 · p. 20

C.5 PAIRWISE PREFERENCE JUDGE

  • The pairwise judge compares two assistants’ replies against a user’s stated preference and returns a brief reason with an A-or-B verdict.The template asks which assistant better fits the preference and requires the exact REASON and VERDICT format.Source: §C.5 · p. 22

D MULTI-TURN TUTORING PROMPTS

  • Tutoring prompts assign tutor turns as assistant messages and student replies as user messages, while the router receives tutor context without persona descriptions.The prompt templates cover the tutor, distillation teacher, student simulator, and judges.Source: D MULTI-TURN TUTORING PROMPTS · pp. 22–23

D.1 TUTOR AND PERSONA DESCRIPTIONS

  • All tutor policies share a brief, conversational teaching prompt, while persona descriptions are used by the distillation teacher and persona-prompting baselines, not learned heads.The described styles contrast clear direct instruction with interactive back-and-forth discussion.Source: §D.1 · p. 23

D.2 DISTILLATION TEACHER

  • The distillation teacher uses the target persona and student reaction while omitting the sampled tutor response, prompting a fresh answer to the original student message.Persona distillation omits reaction text, whereas feedback imitation uses persona- and reaction-conditioned context to generate a complete tutor turn as its target.Source: §D.2 · p. 23

D.3 STUDENT SIMULATOR

  • The student simulator models direct-instruction or interactive preferences, with reaction style fixed within each dialogue and preference-conditioned replies used for training and evaluation at specified stages.For head training, the opening receives a neutral simulated reply; in preference-switching evaluation, the simulator prompt changes beginning with the reply to the switch turn.Source: §D.3 · pp. 23–24

D.4 TRAINING AND EVALUATION JUDGES

  • Training gates use objection and satisfaction prompts without hidden preference or reaction style, while evaluation adds fact-checking and preference-fit judges.The fit judge compares complete tutor dialogues using student reactions, counting complaints and pushback against a tutor and engaged, satisfied replies in its favor.Source: §D.4 · pp. 24–25

E SUPPLEMENTARY PREFERENCE RESULTS

  • Supplementary preference results cover preference learning, composition, and preference switching, with win rates assigning half credit to ties and intervals based on resampling.The supplementary section describes detailed results for these evaluations.Source: E SUPPLEMENTARY PREFERENCE RESULTS · p. 26

E.1 OVERALL PREFERENCE FIT

  • Table 7 reports preference-fit judgment counts and confidence intervals underlying Figure 2; win rates give ties half credit and intervals use resampling.The table caption specifies percentages and reports wins, ties, and losses from PLF’s perspective; it does not provide cell values here.Source: Table 7 · p. 26, §E.1 · p. 26, E SUPPLEMENTARY PREFERENCE RESULTS · p. 26

E.2 HUMAN EVALUATION

Human judgments show the same overall preference for PLF over persona prompting as the model judge. In tutoring, PLF improves preference fit over persona prompting and persona distillation, with higher satisfaction and fewer judged factual errors than persona prompting.

E.2 HUMAN EVALUATION

  • Human majority judgments favored PLF over persona prompting and agreed with the model judge on approximately 76% of single-turn and 73% of tutoring comparisons.The human evaluation sampled 96 single-turn and 100 tutoring comparisons.Source: §E.2 · p. 26, Table 8 · p. 27

E.3 MULTI-TURN PREFERENCE FIT AND TUTORING QUALITY

  • PLF improves tutoring preference fit over persona prompting and persona distillation for both direct-instruction and interactive students.The comparisons use the same 500 test openings for each student type.Source: §E.3 · p. 27, Table 9 · p. 27
  • Against persona prompting, PLF raised satisfaction 2.3 percentage points (95% CI [1.1,3.4]) and lowered flagged factual errors by 1.0 point (paired difference −1.0, 95% CI [−2.0,−0.1]).Satisfaction counts replies not flagged as objections; factual errors count tutor turns flagged by the fact-checking judge.Source: §E.3 · p. 27, Table 10 · p. 27

E.4 SINGLE-TURN COMPOSITION

PLF composes independently trained persona heads for paired preferences. Results vary by pair and whether routing errors are included.

E.4 SINGLE-TURN COMPOSITION

  • Composition coefficients were fixed before test evaluation, with voice strength for thorough pairs selected from {1, 0.5, 0.25} on 50 validation requests.Validation compared composition rules on 100 paired-preference requests against the frozen base prompted with both personas.Source: §E.4 · p. 28, Table 11 · p. 28
  • On 222 correctly routed requests, the mixture’s pooled win rate was 71% [66,77] against the requested voice head and 58% [53,64] against the requested depth head.Warm–concise benefited most from combining both behaviors; impersonal–concise showed no clear advantage over either constituent.Source: §E.4 · p. 28, Table 12 · p. 28
  • Across all 247 paired-preference requests, impersonal–thorough retained an advantage over both heads and warm–concise beat the warm head, while 56% [44,68] against paired-persona prompting was inconclusive.Including routing errors reduced the warm–concise gain against the concise head to an interval reaching parity.Source: Table 13 · p. 28, §E.4 · p. 28

E.5 WITHIN-SESSION PREFERENCE SWITCHING

The frozen router, trained only on fixed-preference dialogues, improves preference fit over fixed heads and the base in both switch directions. Satisfaction rebounds once PLF can respond to the new preference, although selection of the new head can be delayed or absent.

E.5 WITHIN-SESSION PREFERENCE SWITCHING

  • The frozen router improves preference fit over each fixed head and the base in both switch directions, without additional training for switching.The oracle switches at turn j; the student’s reply first reflects the new preference at j, so PLF can react at j + 1.Source: §E.5 · p. 29, Table 14 · p. 29
  • PLF achieves the highest whole-lesson satisfaction among evaluated non-oracle policies, and satisfaction rebounds at the first turn that can respond to the new preference.Table 15 reports overall satisfaction and turn buckets aligned to the switch.Source: §E.5 · p. 29, Table 15 · p. 29
  • PLF selects the newly matching head at the first opportunity in 39.8% of dialogues; 338 of 1,000 dialogues end without such a selection.The mean additional delay is 0.65 turns conditional on selecting the new head; first-opportunity switching is more frequent toward interactive guidance.Source: §E.5 · p. 29

F.1 GATE IMPLEMENTATION AND TRAINING DYNAMICS

The appendix specifies response- and turn-level rules for gating training updates, then compares gated and ungated PLF. Gating improves held-out preference fit in single-turn requests.

F.1 GATE IMPLEMENTATION AND TRAINING DYNAMICS

  • Single-turn responses receive zero loss when the critic requests no changes or provides no usable feedback; tutoring turns are masked only after all three checks accept them.A failed judge call leaves a tutoring turn eligible for training, while simulator failures exclude it.Source: §F.1 · pp. 29–30
  • Switching tables report satisfaction by switch direction and how quickly PLF selects the new head after a preference switch.The oracle anticipates the switch; PLF first observes evidence after generating turn j.Source: Table 16 · p. 30, Table 17 · p. 30
  • Masked tokens remain in Eq. 2's denominator, so gating changes both which responses train and the active loss scale; batches with no positive gates are skipped.Source: §F.1 · p. 30
  • Tutoring satisfaction and masking rates are summarized in 20-cycle blocks, with 256 tutor turns per head per cycle.Source: §F.1 · p. 30, Table 18 · p. 30

F.2 SINGLE-TURN: FIXED-ROUTER COMPARISON

  • The single-turn ablation compares gated and ungated PLF at step 60 on the same 499 held-out requests, with one training run per variant.Over steps 41–60, 25–50% of gated tokens per head carry loss, compared with essentially all valid tokens without gating.Source: §F.2 · pp. 30–31
  • Gating improves held-out preference fit for single-persona and paired requests, with clear single-persona gains for warm and impersonal requests.Critic-approval rates remain similar between runs, so the preference gains are not reflected in approval frequency alone.Source: Table 19 · p. 31, §F.2 · p. 31

F.3 TUTORING: MATCHED-HEAD COMPARISON

Matched-head tutoring comparisons find that gating improves interactive guidance, while a routed single-turn comparison favors forward KL overall. The KL advantage varies by requested preference.

F.3 TUTORING: MATCHED-HEAD COMPARISON

  • The tutoring comparison evaluates matched heads on 500 held-out openings with both student types, excluding routing; each variant uses one training run.Paired eight-turn lessons share their first student reply and sampling seeds; the ungated variant trains on every valid turn.Source: §F.3 · p. 31
  • Gating improves preference fit and satisfaction for interactive guidance, while the direct-instruction comparison is near parity.The gated head asks questions more often and receives fewer repetition complaints, with improvements in both no-objection and strict satisfaction.Source: Table 20 · p. 31, §F.3 · p. 31, Table 21 · p. 32
  • Held-out preference and satisfaction gains persist after the early training-satisfaction gap closes, while the runs converge on strict satisfaction after cycle 50.Gating leaves rollout and teacher computation unchanged; wall-clock times are 5 h 19 min and 5 h 20 min.Source: §F.3 · p. 32

G.1 PROTOCOL AND PREFERENCE FIT

  • Forward KL achieves a 55% win rate [52,59] over reverse KL across 499 routed requests, with strongest gains for impersonal-only and impersonal–thorough requests.Pooling overlapping preference groups gives 64% [59,70] for impersonal, 59% [53,65] for thorough, 51% [45,57] for warm, and 49% [43,55] for concise.Source: Table 22 · p. 32, §G.1 · p. 32

G.2 TEACHER AND RESPONSE DIAGNOSTICS

Teacher diagnostics and generated responses show distinct probability allocations and persona behaviors under forward- and reverse-KL distillation, despite similar top-token agreement.

G.2 TEACHER AND RESPONSE DIAGNOSTICS

  • Reverse-KL heads have lower token entropy, closer to the teacher's, while top-token agreement is similar; forward-KL heads retain more teacher hedge-word probability, especially for impersonal and concise personas.The diagnostics use individual heads on persona-assigned requests and compare them with the reconstructed frozen teacher on observed prefixes.Source: §G.2 · p. 33, Table 23 · p. 33
  • Forward-KL impersonal responses contain 0.65 versus 0.51 hedge expressions per 100 words, accompanying that persona's preference advantage.Source: §G.2 · p. 33
  • Reverse-KL thorough responses have a median length of 1,250 versus 1,114 words without a corresponding preference advantage; concise-only outputs share an eight-word median while judgments favor reverse KL.Source: §G.2 · p. 33
  • The analyses suggest that distillation objective shapes persona expression through probability allocation beyond the top token.Source: §G.2 · p. 33

Related papers

Loading 2610.38561v1…