PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Natural Language Feedback for Personalized Question Answering

Alireza Salemi, Hamed Zamani

arXiv:2508.10695v2cs.CLcs.AIcs.IR

TL;DR

Scalar rewards provide coarse feedback for personalized question answering, offering little direction on how to improve responses. VAC trains with natural-language feedback grounded in user information and reports significant average gains on LaMP-QA, though Art & Entertainment is an exception.

  • Problem

    Scalar rewards provide coarse overall-quality signals without specifying how to improve a response, which can slow convergence and increase training cost.

    Source: §1 · p. 1, §4 · p. 3

  • Method

    VAC alternates between optimizing a user-conditioned feedback model and training the policy model on feedback-refined responses so the policy can generate without feedback at inference.

    Source: §4 · p. 4, Optimizing the Policy Model · p. 5, §6 · p. 10

  • Results

    VAC statistically significantly outperforms all baselines on average across LaMP-QA and reports a 13.6% relative improvement over the non-personalized baseline, though it does not outperform baselines on Art & Entertainment.

    Source: §1 · p. 2, Comparison with the Baselines · p. 7

  • Takeaways & Limitations

    Natural-language feedback preserves feedback structure and semantics, supporting finer-grained alignment between model behavior and user intent.

    Source: §6 · p. 10

    The role of natural-language feedback for partially observed user profiles and bias mitigation remains an open direction.

    Source: §6 · p. 11

Abstract

from arXiv · show

Personalization is crucial for enhancing both the effectiveness and user satisfaction of language technologies, particularly in information-seeking tasks like question answering. Current approaches for personalizing large language models (LLMs) often rely on retrieval-augmented generation (RAG), followed by reinforcement learning with scalar reward signals to teach models how to use retrieved personal context. We believe that these scalar rewards sometimes provide weak, non-instructive feedback, limiting learning efficiency and personalization quality. We introduce VAC, a novel framework for personalized response generation that replaces scalar rewards with natural language feedback (NLF) that are generated conditioned on the user profiles and the question narratives. NLF serves as a rich and actionable supervision signal, allowing the policy model to iteratively refine its outputs and internalize effective personalization strategies. Training alternates between optimizing the feedback model and fine-tuning the policy model on the improved responses, resulting in a policy model that no longer requires feedback at inference. Evaluation on the LaMP-QA benchmark that consists of three diverse domains demonstrates consistent and significant improvements over the state-of-the-art results. Human evaluations further confirm the superior quality of the generated responses. These results demonstrate that NLF provides more effective signals for optimizing personalized question answering.

1 Introduction

Scalar rewards offer weak, non-instructive guidance for personalized question answering, while VAC replaces them with natural-language feedback grounded in user information. The paper reports improvements on LaMP-QA and positive human-evaluation results.

1 Introduction

  • Personalization in question answering aims to align responses with users’ intent, background, and preferences, unlike content generation’s focus on users’ writing style and preferences.The introduction frames personalized question answering as information seeking.Source: §1 · p. 1
  • Scalar rewards indicate whether an output is good or bad but not how to improve it, while single-target supervision can miss other acceptable responses and reward-based exploration slows convergence and raises training cost.The paper describes supervised training on a single target as suboptimal and prone to local minima.Source: §1 · p. 1
  • VAC replaces scalar rewards with feedback conditioned on user profiles and question narratives, then trains the policy on feedback-refined responses so it can generate them without feedback at inference.The feedback model is optimized to produce feedback that improves the current policy's responses according to a task-specific metric.Source: §1 · pp. 1–2
  • VAC gains 13.6% over the non-personalized baseline, 3.6% over the best-performing personalized baseline with 1.9× more inference-time efficiency, and 6.0% over scalar-reward reinforcement learning.Human evaluators preferred VAC in 44% of comparisons, tied in 33%, and preferred the state-of-the-art method in 23%.Source: §1 · p. 2

2 Related Work

Prior work covers several approaches to personalization and has explored natural-language feedback in other tasks. This paper applies such feedback to personalized question answering, with user-conditioned automatic feedback and alternating feedback-policy training.

2 Related Work

  • Personalization research spans search, recommendation, and text generation [12] [21] [27] [28], with question answering targeting responses aligned to users’ preferences, background, and prior knowledge [30].Source: Personalization · p. 2

Personalization.

  • Personalized question answering remains relatively underexplored, and, to the best of the authors’ knowledge, LaMP-QA [30] is the only publicly available benchmark for this setting.Prior personalization methods include a RAG framework using user profiles [28], reinforcement learning to incorporate retrieved personal context [27], personalized retriever feedback [25], user-specific fine-tuning [10], tailored prompts [13], and parameter-efficient fine-tuning for personalized generation [33].Source: Personalization · p. 3

Learning from Natural Language Feedback.

  • Earlier natural-language-feedback work includes math and code tasks [3], collaborative inference [15] [22] [34] [37] [38], and prompt optimization at inference.The paper notes that human-written feedback can improve performance substantially in math and code, while feedback generated by other LLMs tends to be less effective.Source: Learning from Natural Language Feedback · p. 3
  • The paper presents VAC as the first attempt to use natural-language feedback for personalization, addressing subjective user preferences with automatically generated user-conditioned feedback and alternating feedback- and policy-model optimization.The method operates in a RAG framework, where retrieval and generation contribute to the models’ performance.Source: Learning from Natural Language Feedback · p. 3

3 Problem Formulation

The formulation conditions personalized response generation on a user profile and a new query. User-specific aspects and narratives support evaluation, but are withheld from the policy model during generation.

3 Problem Formulation

  • A user profile consists of prior questions and their detailed descriptions, and the policy generates a response conditioned on both that profile and the user’s new query.Source: §3 · p. 3
  • User-provided aspects and a question narrative define evaluation requirements, and a metric scores how well the generated response addresses those aspects.The aspects are derived from user-provided requirements and are used exclusively for evaluation, not made available to the policy during generation.Source: §3 · p. 3
  • Training examples include a query, user profile, expected aspects, and narrative; test examples have the same structure, but aspects and narratives are withheld from the policy model.Source: §3 · p. 3

4 The VAC Framework

VAC replaces scalar rewards with natural-language feedback (NLF) to guide response personalization during training. It alternates feedback and policy optimization so the trained policy can generate responses without feedback at inference.

4 The VAC Framework

  • VAC replaces coarse scalar rewards with NLF and alternately optimizes feedback and policy models, training the policy to personalize responses without feedback at inference.The paper notes scalar rewards do not specify how to improve an output, leaving the model to infer adjustments.Source: §4 · p. 3, Optimizing the Policy Model · p. 5

Overview of the Inference Pipeline:

  • At inference, the trained policy receives a question with retrieved documents from the user profile and generates the response through the RAG pipeline.The retrieval step selects K relevant profile documents for the question.Source: Overview of the Inference Pipeline · p. 4

4.1 Feedback Generation & Output Refinement

  • The feedback model conditions NLF on the question, retrieved profile documents, question narrative, and initial response; the policy then revises that response using the feedback.The revision prompt also includes the initially retrieved documents.Source: Feedback Generation · p. 4, Output Refinement · p. 4

4.2 Feedback & Policy Model Optimization

  • Feedback and policy models are optimized iteratively, with the feedback objective seeking guidance that measurably improves response quality and personalization.The feedback objective aims to produce feedback whose application yields improved responses for training the next policy model.Source: §4.2 · p. 4, Optimizing the Feedback Model · p. 4

Optimizing the Feedback Model.

  • The feedback model is trained on feedback candidates whose revised responses score highest under the task-specific evaluation metric.For each example, candidates are sampled at temperature 1 and applied independently before selecting the highest-scoring feedback; supervised fine-tuning trains the feedback model on that selection.Source: Optimizing the Feedback Model · pp. 4–5

Optimizing the Policy Model.

  • The policy is fine-tuned on responses revised using the current feedback model, learning to produce those responses directly from the input without feedback at inference.This objective assumes the feedback-revised response is a better personalized response for the user.Source: Optimizing the Policy Model · p. 5

5 Experiments

Experiments use LaMP-QA, a personalized question-answering benchmark spanning three domains, and score how well responses address personalized aspects. The policy model is limited to one 7-billion-parameter Qwen 2.5 model because extending the experiments exceeds the stated computational budget.

5.1 Experimental Setup

  • LaMP-QA is the publicly available personalized question-answering benchmark used, covering Art & Entertainment, Lifestyle & Personal Development, and Society & Culture.The paper reports dataset statistics for the benchmark in Table 1.Source: Datasets & Evaluation · p. 5, Table 1 · p. 6

Datasets & Evaluation:

  • Each example includes a user’s question history, question narrative, and personalized rubrics; an instruction-tuned 32-billion-parameter Qwen 2.5 model scores addressed aspects from 0 to 2, normalized to [0, 1] and averaged.The authors also conduct a side-by-side human evaluation in one experiment.Source: Datasets & Evaluation · p. 6
  • Experiments use Qwen 2.5 with 7 billion parameters as the policy model; the authors limit evaluation to this single LLM because extending to larger or alternative models would require over 750 GPU hours beyond their budget.The reported experiments used four NVIDIA A100 GPUs and took around 750 GPU hours.Source: Training Configuration · p. 6

Training Configuration:

The setup specifies a shared inference configuration, personalized and non-personalized baselines, and a three-iteration training comparison. PlanPers uses retrieval followed by planning and response generation.

Inference Configuration:

  • All models use an 8192-token input-output limit and temperature 0.1; retrieval uses Contriever to select k=10 user-profile documents.Generation uses nucleus sampling, and VLLM supports inference and deployment.Source: Inference Configuration · p. 6

Training Configuration:

  • The evaluation table reports LaMP-QA test-set performance and marks significant differences from the best baseline with † at p<0.05.The cited training passage notes diversity among valid personalized outputs and risks misalignment with real user expectations.Source: Training Configuration · p. 6, Table 2 · p. 7

Baselines:

  • The baselines include a question-only LLM, RAG personalization, random-profile RAG, PlanPers, and offline RL RAG personalization.Offline RL selects the highest-scoring of 16 candidates per question using the downstream task metric.Source: Baselines · pp. 6–7
  • PlanPers retrieves profile information, creates a high-level response plan, then generates the personalized answer using the plan, retrieved documents, and question.Figure 4 provides its prompt.Source: Figure 4 · p. 6, Baselines · p. 7
  • The offline RL baseline trains for three iterations under VAC's configuration and uses the best checkpoint; baselines otherwise share VAC's evaluation conditions.The shared conditions include retrieval model, document count, generation temperature, input and output lengths, and training budget.Source: Baselines · p. 7

5.2 Empirical Results

On LaMP-QA, VAC statistically significantly outperforms all baselines in average performance, with gains in two of the three personalized question-answering tasks.

5.2 Empirical Results

Comparison with the Baselines:

VAC provides stronger personalization performance than the baselines with runtime comparable to the most efficient personalized approaches. Jointly updating the feedback model outperforms freezing it, while gains plateau after two iterations.

Comparison with the Baselines:

  • VAC provides superior personalization performance with runtime costs comparable to the most efficient personalized baselines.PlanPers is nearly twice as slow as VAC and yields significantly lower performance; RAG-based methods have higher runtime than the non-personalized LLM because of retrieval overhead.Source: Comparison with the Baselines · pp. 7–8
  • Jointly updating the feedback model and policy model outperforms keeping feedback frozen across all datasets.The feedback model is optimized to match the evolving policy and generate more effective, informative feedback.Source: Effect of Optimizing Feedback Model · p. 8

Effect of Optimizing Feedback Model:

  • VAC performance improves during the first two iterations, then plateaus in the third, with the plateau more pronounced when feedback is untrained.The passage identifies joint optimization of policy and feedback models as important for maximizing effectiveness.Source: Effect of Optimizing Feedback Model · p. 8

Effect of Feedback Model Size:

Larger feedback models are associated with better policy performance, while supervised fine-tuning performs on par with or better than DPO in nearly all reported cases.

Effect of Feedback Model Size:

  • Larger feedback models consistently lead to better policy-model performance in a single training iteration.The comparison uses instruction-tuned Qwen 2.5 models with 1.5, 3, and 7 billion parameters.Source: Effect of Feedback Model Size · p. 8

Effect of Optimization Method for Policy Model:

  • SFT performs on par with or significantly better than DPO, except in the first Art & Entertainment iteration.The paper hypothesizes that DPO's strong separation of initial and updated outputs may distort the output distribution as the policy's responses grow closer to revisions; SFT accommodates subtler improvements.Source: Effect of Optimization Method for… · pp. 8–9
  • The Figure 8 case study presents an initial response, feedback, and updated response, then compares VAC with PlanPers.Source: Figure 8 · p. 9

Human Evaluation:

Human evaluators preferred VAC to PlanPers, and case studies illustrate how feedback and training updates produced more context-aligned responses.

Human Evaluation:

  • 44% of human-evaluation cases favored VAC over PlanPers, compared with 23% favoring PlanPers and 33% judged ties.Annotators compared responses using rubric aspects and the question narrative.Source: Figure 9 · p. 10, Human Evaluation · p. 10

5.3 Case Study

  • For the turkey-stock question, feedback targeted omitted stovetop and bone-type context, and the updated response added doneness cues, monitoring advice, and estimated cooking times.The revision also organized the advice into clear sections.Source: §5.3 · p. 10
  • For exterior-trim refinishing, VAC addressed the user's material, paint, durability, and climate concerns more directly than PlanPers, with higher scores across three rubric dimensions.The cited dimensions are Material Compatibility, Paint Recommendation, and Durability/Longevity.Source: §5.3 · p. 10

6 Conclusions and Future Work

VAC replaces scalar rewards with user-grounded natural-language feedback in an iterative training loop, and the paper reports consistent benchmark gains and human preference. The authors identify broader tasks, model types, and feedback robustness as directions for future work.

6 Conclusions and Future Work

  • VAC replaces scalar rewards with natural-language feedback grounded in user profiles and question narratives, alternating feedback generation with policy refinement so the policy needs no feedback at inference.The conclusion describes this design as amortizing feedback cost into training.Source: §6 · p. 10
  • On LaMP-QA, VAC consistently outperformed existing personalized and non-personalized baselines and was preferred by human evaluators.Source: §6 · p. 10
  • Future work includes feedback over reasoning traces, broader personalization tasks, different LLM classes, and interactions with tool use and agentic workflows.Source: §6 · p. 10
  • Robustness under noisy or partially observed user profiles and feedback's role in mitigating personalization bias remain open directions.Source: §6 · p. 11
Loading 2508.10695v2…