PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubric-Aware On-Policy Self-Distillation for LLM Personalization

Yilun Qiu, Xiaoyan Zhao, Chengbing Wang, Cilin Yan, Rui Zu, Wanyang Zhang, Xiaolong Jiang, Jiayin Cai, Yang Zhang

arXiv:2609.35262v1cs.CL

TL;DR

Personalized question answering needs to address user-specific information needs, but existing rubric-guided methods provide limited guidance for realizing rubric requirements during generation. GRASP distills rubric-conditioned teacher distributions into a rubric-free student and filters inadequate teacher outputs; it achieves state-of-the-art performance across backbones on LaMP-QA.

  • Problem

    Existing rubric-guided personalization provides limited guidance on how to realize user-specific requirements during generation.

    Source: §1 · p. 2

  • Method

    GRASP aligns a rubric-free student with a frozen rubric-informed teacher along student-generated trajectories and uses Rubric-based Teacher Validation to retain adequately covering teacher outputs.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    GRASP consistently achieves state-of-the-art performance on LaMP-QA across four backbones; on Gemma2-9B, its average relative improvement over the strongest baseline is 14%.

    Source: §5.2 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    The findings demonstrate the value of fine-grained distributional supervision for personalization.

    Source: §6 · p. 10

    Training data has no gold reference responses, and inference receives only the question and profile, without rubric aspects.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

LLM personalization aims to generate responses aligned with individual users' preferences and needs. User-specific rubrics make these expectations explicit, providing direct supervision on what a satisfactory answer should cover. Existing rubric-guided approaches, however, exploit such guidance only at a coarse granularity, either by using rubrics to supervise the prediction of relevant aspects for subsequent generation or by reducing aspect coverage to a single response-level reward for reinforcement learning. This leaves a gap between specifying what a personalized answer should contain and teaching the model how to generate it. To bridge this gap, we propose GRASP, a rubric-aware on-policy self-distillation framework for LLM personalization that turns user-specific rubric aspects into fine-grained, token-level supervision. Specifically, GRASP pairs a rubric-free student with a rubric-informed teacher that additionally receives the target user-specific rubrics. By aligning their next-token distributions along on-policy trajectories generated by the student, GRASP transfers the teacher's rubric-conditioned guidance into the student, translating user-specific semantic requirements into dense token-level supervision. Since rubric-informed teachers can still produce inadequate supervision, we further introduce Rubric-based Teacher Validation (RTV), which retains only instances where the teacher sufficiently covers the target aspects, improving both supervision quality and training efficiency. Experiments on the LaMP-QA benchmark for personalized question answering demonstrate that GRASP achieves state-of-the-art performance across multiple backbones, supporting the effectiveness of rubric-guided token-level supervision for personalization. To ensure reproducibility, our code is available at https://github.com/SnowCharmQ/GRASP.

1 INTRODUCTION

Personalized question answering must address the aspects that matter to each user, not merely provide factually correct answers. GRASP uses rubric-informed on-policy self-distillation and teacher validation to provide token-level guidance, and achieves state-of-the-art performance across backbones on LaMP-QA.

1 INTRODUCTION

  • The growing use of LLMs in information seeking and decision support motivates personalization for users with diverse preferences and needs.In question answering, people asking the same question may seek different information.Source: §1 · p. 1
  • Personalized answers must cover user-relevant aspects, since users asking the same question may prioritize different information.User-specific rubrics formalize what a satisfactory answer should address.Source: §1 · p. 1
  • Existing methods use rubrics coarsely: PlanPers predicts aspects for a planner, while IAP uses a scalar coverage reward for reinforcement learning.Planner supervision does not directly supervise the generator, and the scalar reward gives limited feedback on token-level decisions.Source: §1 · p. 2
  • GRASP pairs a rubric-free student with a rubric-informed teacher and distills teacher distributions along student-generated trajectories.The student receives dense, rubric-conditioned token-level supervision during training.Source: §1 · p. 2
  • Rubric-based Teacher Validation retains teacher outputs that adequately cover target aspects, improving distillation reliability and sample efficiency.The teacher may provide inadequate coverage even when conditioned on the target rubrics.Source: §1 · p. 2
  • GRASP achieves state-of-the-art performance on LaMP-QA across different backbone models.The benchmark spans three diverse domains.Source: §1 · p. 2

2 RELATED WORK

Prior personalization research has focused more on generating content in a user's style than answering according to their information needs. On-policy self-distillation remains unexplored for personalization, which this work brings into personalized question answering.

2 RELATED WORK

  • Personalized question answering requires inferring what a user needs to know, unlike personalized content generation, which imitates how they write.LaMP-QA pairs questions with user profiles and rubric aspects specifying satisfactory-answer information needs.Source: §2 · pp. 2–3
  • IAP uses intent articulation and reinforcement learning with rubric-derived rewards, while VAC uses learned natural-language critiques and revised responses.Both approaches build on LaMP-QA for personalized question answering.Source: §2 · p. 3
  • On-policy self-distillation remains unexplored for personalization, and this work applies it to transfer user-specific rubric guidance.In OPSD, student and teacher are instantiated from one model under different contexts, with privileged information available only to the teacher.Source: §2 · p. 3

3 PRELIMINARY

The preliminary section formulates personalized question answering around user profiles and rubric aspects, then introduces on-policy self-distillation as the method's foundation. Rubrics guide training, while inference uses only the question and profile.

3.1 PROBLEM FORMULATION

  • Personalized response quality is characterized by coverage of user-specific rubric aspects, each describing a criterion the response should address.The rubric-based score measures how well a response addresses an aspect.Source: §3.1 · p. 3
  • PQA training uses questions, user profiles, and rubric aspects without gold reference responses, and inference receives only the question and profile.Rubric aspects characterize the information a satisfactory response should cover for each user.Source: §3.1 · p. 3

3 PRELIMINARY

  • Figure 1 depicts rubric-aware teacher distributions guiding the student on its own trajectories, alongside validation that retains outputs with adequate rubric coverage.The figure presents these as the method's two components.Source: Figure 1 · p. 4

3.2 ON-POLICY SELF-DISTILLATION

  • On-policy self-distillation uses student-generated responses and minimizes per-token divergence between teacher and student distributions along those trajectories.The student and teacher are the same language model conditioned on different contexts; privileged information is given to the teacher.Source: §3.2 · p. 4

4 METHODOLOGY

GRASP treats user-specific rubrics as privileged information, transferring their guidance to a rubric-free student through on-policy self-distillation and filtering teacher signals with RTV.

4.1 RUBRIC-AWARE ON-POLICY SELF-DISTILLATION

  • The student receives the question and user profile, while a frozen instance of the same model also receives target rubric aspects as the teacher.The student is trainable; the rubric-informed teacher remains fixed.Source: §4.1 · pp. 4–5
  • Responses sampled from the student provide shared prefixes on which the student and rubric-informed teacher produce directly comparable next-token distributions.Because the student conditions only on the question and profile, its sampled responses use the same information setting as inference.Source: §4.1 · p. 5
  • GRASP aligns the full-vocabulary next-token distributions at each decoding step using forward KL, transferring rubric-conditioned token preferences to the student.The divergence is between next-token distributions over the vocabulary.Source: §4.1 · p. 5

4.2 RUBRIC-BASED TEACHER VALIDATION

  • RTV scores rubric-informed teacher responses for aspect coverage and retains only instances meeting a threshold; the main setting uses τ = 1, requiring full coverage.The authors apply RTV once to the original training set before distillation.Source: §4.2 · pp. 5–6

4.3 TRAINING OBJECTIVE

  • The overall objective minimizes teacher–student divergence on validated instances and student-sampled responses, with gradients propagated only through the student.Teacher distributions are fixed targets.Source: §4.3 · p. 6

5 EXPERIMENTS

Experiments on LaMP-QA compare GRASP with personalization baselines across four backbones and analyze its validation, teacher-capability, rubric-alignment, and token-distribution mechanisms. GRASP leads the benchmark comparison, while the analyses support reliable, question-aligned rubric supervision as central to its approach.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses LaMP-QA, a personalized long-form question-answering benchmark spanning three categories, and reports per-category scores and their macro average.The categories are Arts & Entertainment, Lifestyle & Personal Development, and Society & Culture.Source: §5.1 · p. 6
  • The study compares profile-free answering, profile retrieval, silver-label fine-tuning, aspect planning, and rubric-reward reinforcement learning baselines.The baseline set includes Non-Perso, RAG, SFT, PlanPers, and IAP.Source: §5.1 · p. 6
  • Each backbone serves as both GRASP's student and teacher, with the student fine-tuned using full-vocabulary forward KL and a separately validated teacher.The student and frozen teacher share initialization; RTV is conducted separately for each backbone.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • GRASP obtains the highest average score on all four backbones, with a 14% average relative improvement over the strongest baseline on Gemma2-9B.Table 1 compares methods across four backbones on LaMP-QA; higher values indicate better performance.Source: Table 1 · p. 7, §5.2 · p. 8

5.3 FRAMEWORK ANALYSES

  • RTV with τ = 1 improves performance across all four backbones, and stricter validation generally helps on Gemma2-9B and Qwen2.5-7B.Lower thresholds show limited, non-monotonic gains; the authors support τ = 1 as the default setting.Source: §5.3 · p. 8, Figure 2 · p. 8
  • Scaling RAG models yields inconsistent gains and can degrade performance, while a larger GRASP teacher offers no clear advantage over self-distillation.The analysis suggests model or teacher capacity alone is not the key to effective personalized question answering.Source: §5.3 · p. 8, Figure 3 · p. 8
  • Shuffling rubric sets across instances substantially reduces performance, supporting the importance of matching rubric aspects to each question and user profile.The shuffled variant preserves the rubric distribution but breaks instance-level alignment.Source: §5.3 · p. 9, Table 2 · p. 9
  • Rubric-induced and training-induced token shifts agree for 79.2% of 14,216 tokens (r = 0.71), and average response-level forward KL falls from 1.101 to 0.444 after training.All 90 analyzed responses move closer to the rubric-conditioned teacher; the matched-prefix setup compares distributions at the same sampled prefixes.Source: §5.3 · p. 9, Figure 4 · p. 9

6 CONCLUSION

The conclusion presents GRASP as a rubric-aware on-policy self-distillation framework for personalized question answering. Across four LaMP-QA backbones, it reports state-of-the-art performance and identifies fine-grained distributional supervision as valuable for personalization.

6 CONCLUSION

  • GRASP aligns a rubric-free student with a frozen rubric-informed teacher on student-generated trajectories to turn user-specific aspects into token-level supervision.Rubric-based Teacher Validation is introduced to improve teacher-guidance reliability.Source: §6 · p. 10

A DATASET DETAILS

This appendix describes LaMP-QA and the baseline methods used for comparison. The benchmark pairs questions with user profiles and rubric aspects, while the baselines represent several approaches to personalization.

A DATASET DETAILS

  • LaMP-QA treats the current question as input, prior questions as the user profile, and narrative-derived key aspects as the evaluation rubric.The benchmark covers Arts & Entertainment, Lifestyle & Personal Development, and Society & Culture.Source: A DATASET DETAILS · p. 13

B BASELINE DETAILS

  • Non-Perso answers without profile information, while RAG adds the top-10 retrieved profile entries as context.These baselines measure general question answering without personalization and gains from incorporating user history, respectively.Source: B BASELINE DETAILS · p. 13
  • SFT selects the highest-scoring response from five candidates for fine-tuning, PlanPers predicts relevant aspects before generation, and IAP optimizes intent-conditioned responses with DAPO.IAP's reward combines rubric-based personalization quality, a contrastive term, and a penalty on excessively long intent reasoning.Source: B BASELINE DETAILS · p. 13

C IMPLEMENTATION DETAILS

The implementation uses rubric-conditioned teacher responses to assess teacher reliability and fine-tunes a student on its own sampled trajectories. Evaluation follows the LaMP-QA generation protocol, with a separate larger-model comparison for teacher capability.

C IMPLEMENTATION DETAILS

  • RTV generates one rubric-conditioned validation response per training instance and averages normalized per-aspect judge scores into a teacher reliability score.The rubric judge is Qwen2.5-32B-Instruct, using the test-set per-aspect scoring procedure.Source: C IMPLEMENTATION DETAILS · pp. 13–14
  • Table 4 compares methods by profile conditioning, backbone training, rubric use, and rubric-supervision granularity, ranking the finest-grained mechanism as #1.Source: Table 4 · p. 14
  • The student and frozen teacher share initialization; the student is fully fine-tuned on on-policy responses using full-vocabulary forward KL at shared prefixes.Training uses a maximum sequence length of 6,144 tokens, including up to 512 completion tokens.Source: C IMPLEMENTATION DETAILS · p. 14
  • Evaluation responses use the top-10 retrieved profile entries, temperature 0.1, top-p 0.95, and a maximum generation length of 2,048 tokens.Source: C IMPLEMENTATION DETAILS · p. 14
  • Teacher-capability analysis pairs each student with a larger model from the same family, testing that model both as a standalone RAG baseline and as a rubric-informed teacher.Source: C IMPLEMENTATION DETAILS · p. 14

D.1 RTV ABLATION AND THRESHOLD ANALYSIS

RTV’s impact depends on the teacher’s reliability distribution: stricter filtering removes more low-reliability supervision when that mass is substantial. Gemma2-9B has a heavier low- and intermediate-reliability tail than the other reported backbones.

D.1 RTV ABLATION AND THRESHOLD ANALYSIS

  • Only 37% of Gemma2-9B instances attain sT = 1, versus 57%, 62%, and 81% for Qwen2.5-14B, Qwen2.5-7B, and Qwen3-4B, respectively.Figure 5 shows Gemma2-9B’s heavier tail in low- and intermediate-reliability regions.Source: §D.1 · p. 14, Figure 5 · p. 15
  • Raising the RTV threshold filters out a larger share of unreliable supervision for Gemma2-9B and is associated with more pronounced performance gains.The other teachers’ reliability scores cluster more at sT = 1, so increasing the threshold changes their retained training sets less and yields smaller incremental gains.Source: §D.1 · pp. 14–15
  • RTV’s effectiveness is closely related to the teacher’s reliability distribution, with stricter filtering particularly beneficial when low-score mass is substantial.Source: §D.1 · p. 15

D.2 TRAINING EFFICIENCY

GRASP’s training finishes in 6.7–11.9 hours across four backbones and adds no inference cost over RAG. The rubric-completeness comparison favors providing the teacher with the full target rubric set over one randomly sampled aspect.

D.2 TRAINING EFFICIENCY

  • Across four backbones, GRASP training takes 217–470 steps and 6.7–11.9 hours, with no additional inference cost over standard RAG.At inference, GRASP uses only the question and retrieved user profile, without rubric aspects or the teacher model.Source: §D.2 · p. 15

D.3 EFFECT OF RUBRIC COMPLETENESS

  • The complete rubric set consistently outperforms a single randomly sampled aspect across all categories and backbones.A single aspect gives the teacher only a partial view of the user’s information needs, whereas the full set supplies complementary information for more comprehensive guidance.Source: §D.3 · pp. 15–16

D.4 TRAINING DYNAMICS

In the LaMP-QA case study, GRASP covers all three rubric aspects while the RAG baseline misses user-specific requirements. Training signals also show rapid convergence and gradual stabilization across backbones.

D.4 TRAINING DYNAMICS

  • During the single epoch, forward KL decreases, student–teacher top-1 agreement increases, and gradient norm drops by about one order of magnitude.Most changes occur within the first 20–30% of training, after which the signals gradually stabilize across backbones.Source: §D.4 · p. 16

F PROMPTS

  • The student and RAG share a prompt with the question and top-10 profile entries, while the teacher additionally receives the user’s elaboration and target rubric aspects.The prompt construction isolates rubric information as the difference between student and teacher distributions.Source: F PROMPTS · p. 16, P1 — Student prompt · p. 18

P3 — Judge

E CASE STUDY

  • The aspect-coverage score rises from 0.33 for RAG to 1.00 for GRASP, which addresses all three rubric aspects.Figure 7 compares both systems using the same inference-time input; rubric aspects are ground-truth evaluation criteria, not provided during inference.Source: E CASE STUDY · p. 16, Figure 7 · p. 17
Loading 2609.35262v1…