PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning What to Keep: Transfer-Aware Personal Skill Learning from User Feedback

Li Wang, Chen Zhang, Yirong Yang, Xingyu Chen, Jinkun Zhao, Tianhao Peng, Wenjun Wu

arXiv:2610.19380v1cs.CL

TL;DR

Mixed user feedback combines reusable preferences with request-specific corrections, so retaining all of it risks negative transfer. SkillGrain decomposes feedback into atomic rules and learns which to retain, using fine-grained rewards to train skill induction and downstream generation. Across three base models and in- and out-of-domain settings, it achieves the highest overall average score on each model.

  • Problem

    User feedback can mix transferable preferences with request-specific requirements, making it important to identify which individual components generalize to future requests.

    Source: §1 · p. 1

  • Method

    SkillGrain decomposes mixed feedback into atomic rules, predicts transferability, separately credits rule content and retention, and jointly optimizes skill induction and downstream application.

    Source: §1 · p. 2

  • Results

    Across in-domain and out-of-domain settings, SKILLGRAIN achieves the highest overall average score on each of three base models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The five-task writing benchmark tests whether feedback-derived skills preserve reusable preferences without carrying over request-specific information.

    Source: §1 · p. 2

    The benchmark includes LLM-synthesized user profiles, and experiments focus on 1.7B–4B models and writing tasks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

User feedback enables LLMs to refine reusable personalized skills, but often mixes persistent preferences with request-specific corrections. Indiscriminately retaining both risks negative transfer, motivating selective retention at the level of individual requirements. We frame this challenge as selective transfer over atomic rules and propose SkillGrain, a framework for learning reusable personalized skills from user feedback. SkillGrain decomposes feedback into independently actionable rules, predicts their transferability, and retains only those suitable for future requests. To learn these decisions, we combine rubric-guided skill rewards with fine-grained credit assignment that separately evaluates rule content and retention decisions. We further jointly optimize skill induction and skill-conditioned generation through reinforcement learning, ensuring that learned skills support downstream personalization. We construct a benchmark spanning five personalized writing tasks to evaluate cross-request generalization. Experiments across three base models in both in-domain and out-of-domain settings show that SkillGrain achieves the best overall average performance among competing methods.

1 Introduction

User feedback can mix persistent preferences with request-specific details, so retaining it indiscriminately risks inappropriate constraints and negative transfer. SkillGrain selectively retains atomic rules, and the paper reports the highest average score on each tested model.

1 Introduction

  • Mixed feedback requires identifying which individual requirements transfer across future requests, because retaining task-specific details can introduce inappropriate constraints and negative transfer.Style or organization can reflect persistent preferences, while specific entities or facts are often tied to the current request.Source: §1 · p. 1
  • The five-task writing benchmark tests whether feedback-derived skills preserve reusable preferences without carrying over request-specific information.The evaluation covers three base models and both in-domain and out-of-domain settings.Source: §1 · p. 2
  • SKILLGRAIN decomposes feedback into atomic rules, predicts each rule’s transferability, and separately credits rule content and retention while jointly optimizing skill induction and downstream use.The framework uses rubric-guided skill rewards and fine-grained credit assignment.Source: §1 · p. 2
  • SKILLGRAIN achieves the highest overall average score on each of three base models across in-domain and out-of-domain settings.The benchmark spans five personalized writing tasks.Source: §1 · p. 2

2 Related work

Prior work studies reusable agent skills and user personalization through distinct approaches. This paper extends both directions by distilling mixed user feedback into reusable personalized skills.

2.1 Skill Evolution in LLM Agents

2.2 Large Language Model Personalization

  • Personalization work infers personas, learns shared and user-specific preferences, and organizes interaction histories into memories, including distinctions between stable and context-dependent preferences (Liu et al., 2025; Balepur et al., 2025; Choi et al., 2025; Ma et al., 2026; Zhang et al., 2025; Sun et al., 2026; Gao et al., 2026; Zhang et al., 2026).The paper complements these approaches by distilling user feedback into reusable personalized skills.Source: §2.2 · pp. 2–3

3 Task and Benchmark Construction

The paper models personalized skill learning as an interaction process in which feedback updates a skill used for later requests. Its benchmark covers five writing tasks, separating in-domain evaluation from held-out out-of-domain evaluation.

3.1 Task Formulation

  • After each request and feedback, the personalized skill is updated for use on later requests, with the goal of capturing user preferences while avoiding interaction-specific information.The interaction process includes a user’s fixed historical interactions and a maintained skill.Source: §3.1 · p. 3

3.2 Benchmark Construction

  • Five writing tasks are split into three training and in-domain tasks and two held-out out-of-domain tasks; user-history-based queries undergo automatic and LLM quality checks.Structured revision targets distinguish reusable preference revisions from interaction-specific requirements, using a transferability label of 1 or 0, respectively.Source: §3.2 · p. 3

4 Method

SkillGrain treats personalized skill learning as selective transfer: it decomposes feedback into atomic rules, predicts which rules should persist, and jointly trains skill updates and personalized responses. Its training uses rubric-guided rewards and fine-grained credit assignment to distinguish rule quality from retention decisions.

4.1 Transfer-Aware Atomic Skill Induction

  • Feedback is decomposed into atomic rules, each with a transferability decision; only rules predicted to transfer are appended to the existing skill.If no rule is predicted transferable, the existing skill remains unchanged.Source: §4.1 · pp. 3–4

4.2 Rubric-Guided Rewards

  • Rubric-guided skill rewards assess correct updates to transferable preferences, penalize retained interaction-specific information, and evaluate whether an overall update is warranted.The skill reward evaluates skill-update trajectories, while a separate response reward supervises skill-conditioned generation.Source: §4.2 · pp. 4–5

4.3 Fine-Grained Advantage Assignment

  • Fine-grained credit assignment separately evaluates rule-content correctness and transferability decisions for reliably matched generated rules.Verified fields receive length-normalized local signals; unverifiable tokens retain the trajectory-level advantage, and confidently identified omitted transferable targets receive an omission penalty.Source: §4.3 · p. 5

4.4 Joint GRPO Training

  • SkillGrain jointly optimizes skill-update and response trajectories with GRPO, using separate rubric-based rewards and independently normalized advantages.At inference, the learned user skill guides personalized responses without reward computation or group sampling.Source: §4.4 · p. 5

5 Experiments

Experiments compare SkillGrain with inference-only and training-based baselines across five writing tasks, then test its components, adaptation rounds, data scale, rubric transfer, and sensitivity to local-credit weight.

5.1 Experiment Setup

  • Across five writing tasks, evaluation uses held-out examples from three training tasks in-domain and two unseen tasks out-of-domain.Baselines include inference-only personalization approaches and training-based SFT, self-training, and RL methods.Source: §5.1 · pp. 5–6

5.2 Main Results

  • SkillGrain achieves the highest average performance across all three backbones, exceeding the strongest baseline by 1.61, 1.13, and 1.32 points, respectively.The gains correspond to Qwen2.5-3B-Instruct, Qwen3-1.7B, and Qwen3-4B.Source: §5.2 · p. 6, Table 1 · p. 7

5.3 Ablation Study

  • Removing atomization or skill-update RL causes the largest performance drops, while the other ablations also reduce performance.The ablations replace rubric-based skill rewards, atomic updates, local advantage assignment, or skill-update RL with alternative or removed components.Source: §5.3 · p. 6, Table 2 · p. 8

5.4 Effect of Adaptation Rounds

  • After the second adaptation round, SkillGrain maintains the best performance across all three backbones, and usually further enlarges its advantage over competing methods.Most methods improve as additional feedback becomes available.Source: §5.4 · pp. 6–7, Figure 3 · p. 8

5.5 Effect of Training Data Scale

  • Performance generally improves as training data increases, with gains gradually saturating as the full training set is approached.The comparison uses different proportions of available training data.Source: §5.5 · p. 7, Figure 3 · p. 8

5.6 Transfer to Alternative Rubrics

  • On four unseen writing preferences, SkillGrain achieves the best or tied-best average performance across all three backbones, providing preliminary evidence of transfer beyond training rubrics.The evaluation uses 40 held-out request pairs and deterministic rule-adherence checks without further training.Source: §5.6 · pp. 7–8, Table 3 · p. 8

5.7 Sensitivity to Local-Credit Weight

  • Across λ values of 0.5, 1.0, and 1.5, performance changes modestly, with no setting consistently best across backbones.The default λ = 1.0 remains competitive, indicating robustness to moderate weight variations.Source: §5.7 · p. 8, Figure 4 · p. 8

6 Conclusion

The conclusion presents SkillGrain as a method for learning personalized writing skills from mixed-scope feedback. It combines atomic transfer decisions and rubric-guided learning, with experiments across multiple backbones demonstrating effectiveness.

6 Conclusion

  • SkillGrain is proposed to handle feedback containing both transferable and task-specific information.This is the paper's stated focus for feedback-driven personalized skill learning in writing.Source: §6 · p. 8
  • SkillGrain addresses writing-task feedback that combines transferable preferences with task-specific information.The framework decomposes feedback into atomic rules, learns their transferability, and uses rubric-based rewards with fine-grained advantage assignment to optimize skill updating and use.Source: §6 · p. 8
  • The benchmark spans five writing tasks, and experiments across multiple backbones demonstrate the approach's effectiveness.The conclusion describes joint optimization of skill updating and utilization.Source: §6 · p. 8

Limitations

The study identifies limits in benchmark realism and experimental scope.

Limitations

  • Some benchmark user profiles are synthesized by LLMs and may not capture the diversity and complexity of real preferences.This constrains how closely the benchmark represents real users.Source: Limitations · p. 9
  • Experiments mainly cover models in the 1.7B–4B range and writing-oriented tasks because of computational constraints.The study does not report evaluation on larger models or more complex task domains.Source: Limitations · p. 9
  • Evaluation on larger models, real-world users, and more complex task domains remains future work.These are the authors' stated directions for extending the study.Source: Limitations · p. 9

Ethical Considerations

The paper notes that personalized skills and source corpora can carry privacy and safety risks, and that its non-leakage rubric does not establish protection against them.

Ethical Considerations

  • Personalized skills may retain or reproduce user-specific details from interaction histories, while source corpora may contain identifiable or objectionable text.These risks are relevant to releasing derived data or deploying the method with real users.Source: Ethical Considerations · p. 9
  • Incorrectly transferring a preference or factual detail to a new request may produce misleading personalized responses.The concern includes both preference and factual carryover.Source: Ethical Considerations · p. 9
  • The non-leakage rubric measures selected forms of factual carryover, but does not establish privacy protection or content safety.The paper says these risks require separate review before release or deployment with real users.Source: Ethical Considerations · p. 9

A.1 Benchmark Construction

The benchmark combines five personalized writing tasks with rubric-based evaluation and structured feedback to study skill learning across interactions. The supplied subsections also describe fine-grained credit assignment and the shared training and comparison setup.

A.1 Benchmark Construction

  • Five writing tasks use historical examples grouped by user or author, with synthesized interaction queries and automatic and LLM-based checks to filter duplicates and inconsistent instances.The joint training pool contains 1,498 instances from Email, Topic Writing, and Yelp; the 998-instance test split covers all five tasks, with Abstract and Product Review held out from training.Source: §A.1 · p. 10

A.2 Rubric Scoring

  • Response rewards average four normalized dimensions: length preference, first-person usage, expression preference, and non-leakage.Length and first-person scores use piecewise-linear interval scoring, granting full credit within a preferred range and linearly penalizing deviations.Source: §A.2 · pp. 11–12

A.3 Rubric-Grounded Feedback Construction

  • Deterministic rubric rules identify reusable-preference and interaction-specific revision targets, which a fixed Qwen3-4B model verbalizes as concise feedback.The verbalizer is instructed to preserve the revision requirements without adding preferences or facts.Source: §A.3 · p. 12

B.1 Fine-Grained Credit Assignment

  • Generated atomic rules are aligned to structured revision targets, while confidently missing transferable targets receive an omission penalty.Ambiguous, conflicting, or unsupported matches abstain from local credit assignment and retain the sequence-level advantage.Source: §B.1 · p. 12

B.2 Training and Comparison Details

  • SKILLGRAIN predicts atom transferability, unlike holistic keep-or-update baselines; its training jointly optimizes skill induction and response generation.All methods other than NO HISTORY use five same-user historical interactions; ATOM-PROMPT applies the atomic protocol without parameter optimization.Source: §B.2 · p. 13

B.3 Dimension-wise Results

Across the reported evaluations, SKILLGRAIN leads on length scores, improves non-leakage over OUTCOME-RL, maintains high transferability precision, and receives the highest independent-judge mean score.

B.3 Dimension-wise Results

  • SKILLGRAIN achieves the highest LEN score across all three backbones and improves NL over OUTCOME-RL throughout.The reported pattern reflects better length alignment and reduced inappropriate factual carryover.Source: §B.3 · p. 13, Table 5 · p. 13

B.4 Atomic Transferability Analysis

  • Atomic extraction and transferable-atom selection precision remain high across backbones and both evaluation domains.The authors interpret these results as indicating that the model identifies reusable preferences while limiting inappropriate transfer into persistent skills.Source: §B.4 · p. 13

B.5 Evaluation with an Independent LLM Judge

  • On 200 randomly sampled test instances, GPT-6 Luna assigns SKILLGRAIN the highest mean judge score among the three methods.The sample covers 40 instances per task and evaluates four personalization dimensions.Source: §B.5 · p. 13

B.6 Comparison of Computational Cost

  • Training times are compared for methods requiring optimization using Qwen3-4B on four H100 GPUs.The supplied passages identify the comparison scope but do not report the training-time outcomes.Source: §B.6 · p. 13

C Training Algorithm

The appendix presents the training procedure, prompt conventions, and a two-round case study of atom-based skill updates. It also discloses LLM assistance in refining parts of the text.

C Training Algorithm

  • Algorithm 1 summarizes SKILLGRAIN’s training procedure, while the supplied algorithm fragments show skill and response rollouts and an omission-penalty step.The procedure also computes separate skill and response losses before updating model parameters.Source: C Training Algorithm · p. 13, E Prompts · p. 15

D Use of Large Language Models

E Prompts

  • Except for NO HISTORY, methods share five historical examples, the current request, actor model, and answering prompt; the fixed judge verbalizes editing notes but does not compute rewards.Figure 5 is the common answering prompt, and Figure 8 shows the atom-decomposition prompt whose program concatenates only atoms marked transferable.Source: E Prompts · p. 14, Figure 5 · p. 14, Figure 8 · p. 16

F Case Study

  • In the abstract-writing case, a general preference for useful detail is retained, while request-specific terms are excluded from persistent skill; a later request with only local feedback leaves the skill unchanged.The held-out test response receives 1.000 on length, first-person usage, expression, and non-leakage.Source: Stage LEN FP EXPR NL Mean · p. 17, Table 9 · p. 17
Loading 2610.19380v1…