PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs

Chenxing Wei, Hong Wang, Ying He, Fei Yu, Yao Shu

arXiv:2509.23166v3cs.CL

TL;DR

LLMs can degrade in multi-turn interactions, while prevailing alignment approaches predominantly rely on single-turn data. The paper proposes feedback-driven test-time adaptation and ROSA, reporting improved effectiveness and efficiency over baselines.

  • Problem

    LLMs often degrade in multi-turn interactions, while prevailing alignment methods predominantly rely on single-turn training and evaluation data.

    Source: §1 · p. 1

  • Method

    ROSA operationalizes test-time policy adaptation by using conversational feedback to estimate an optimal policy and steer model parameters in one update.

    Source: §1 · p. 2

  • Results

    ROSA outperforms baseline methods in both effectiveness and efficiency across challenging datasets.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper establishes that ROSA’s gap with user preferences narrows as the number of interaction turns increases.

    Source: §1 · p. 3

    ROSA is currently less effective on tasks heavily bottlenecked by the model’s pre-trained knowledge.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) employ multi-turn interaction as a fundamental paradigm for completing complex tasks. However, their performance often degrades in extended interactions, as they are typically trained on static, single-turn data, which hinders their ability to adapt to real-time user feedback. To address this limitation, we first propose a new paradigm: Test-Time Policy Adaptation for Multi-Turn Interactions (T2PAM), which utilizes user feedback from the ongoing interaction as a reward signal to estimate a latent optimal policy aligned with user preferences, then updates a small subset of parameters to steer the model toward this policy, ultimately enabling efficient in-conversation self-correction. We then introduce Optimum-Referenced One-Step Adaptation (ROSA), a lightweight algorithm that operationalizes T2PAM. ROSA guides the model parameters toward a theoretical optimal policy in a single, efficient update step, avoiding costly iterative gradient-based optimization and minimizing computational overhead. We provide a rigorous theoretical analysis guaranteeing that the policy of ROSA converges to the preference of user as the number of interactions increases. Extensive experiments on challenging benchmark demonstrate that ROSA achieves significant improvements in both task effectiveness and efficiency.

1 INTRODUCTION

The introduction describes a mismatch between single-turn alignment and multi-turn use, where LLM performance can degrade. It proposes feedback-driven test-time adaptation and ROSA, with reported gains in effectiveness and efficiency.

1 INTRODUCTION

  • LLMs often degrade in multi-turn interactions, while prevailing SFT and RLHF approaches predominantly train and evaluate on single-turn data.The introduction links this paradigm mismatch to limits in complex interactions and a gap between benchmark performance and practical utility.Source: §1 · p. 1
  • T2PAM uses conversational feedback as a reward signal to adapt policy during inference, aiming for lightweight, user-specific alignment within each conversation.The target preference may vary across tasks, while the stated objective is a response the user accepts.Source: §1 · p. 2
  • ROSA operationalizes T2PAM by estimating an optimal policy from user feedback and steering model parameters toward it in one update.This avoids costly iterative optimization and is intended to minimize computational overhead.Source: §1 · p. 2
  • The paper establishes a theoretical result that ROSA’s gap with user preferences narrows as interaction turns increase.The introduction also identifies data-collection costs, long-context training, and distribution-mismatch risks as obstacles for multi-turn training strategies.Source: §1 · pp. 2–3
  • ROSA outperforms baseline methods in both effectiveness and efficiency across challenging datasets.The introduction also reports theoretical analysis of ROSA and online parameter updates for rapid preference alignment.Source: §1 · p. 3

2 THE T2PAM PARADIGM

The section diagnoses weak correction across turns and presents T2PAM as online policy adaptation rather than static inference. It frames the paradigm as combining inference-time parameter updates with feedback from the live conversation.

2 THE T2PAM PARADIGM

  • Current models show sharply diminishing accuracy gains across 10 turns, with very few successful corrections after the first attempt on MATH.Figure 3 counts newly solved problems per turn on MATH.Source: §2 · p. 3, Figure 3 · p. 3
  • T2PAM shifts alignment from offline training to inference-time parameter modification, using live conversational feedback for real-time policy updates.The section frames this as combining prompting’s inference-time operation with more direct policy adaptability, without additional offline training cost.Source: §2 · p. 3, Table 1 · p. 4
  • At each turn, the model generates a response and receives a scalar success or failure reward; failure triggers an online update intended to improve the next response.A successful task ends the interaction; the paradigm’s adaptation function updates parameters after failure feedback.Source: §2 · p. 4

3 OPTIMUM-REFERENCED ONE-STEP ADAPTATION

ROSA operationalizes test-time policy adaptation by using feedback to estimate a target policy and moving model parameters toward it in one efficient update. Its steps combine a KL-regularized reward objective, an analytically derived target, and a linearized parameter update.

3 OPTIMUM-REFERENCED ONE-STEP ADAPTATION

  • ROSA uses real-time user feedback to guide an efficient, one-step update toward a theoretical optimal policy, avoiding iterative gradient-based optimization.The method uses a closed-form target and computes parameter updates with Conjugate Gradient.Source: §3 · p. 4

3.1 THE RLHF OBJECTIVE FOR TURN-WISE ADAPTATION

  • The turn-wise RLHF objective maximizes binary feedback reward while using KL regularization to limit divergence from the previous policy.β controls the strength of the KL regularization.Source: §3.1 · p. 4

3.2 FROM THEORETICAL OPTIMUM TO A PRACTICAL ONE-STEP UPDATE

  • ROSA derives a closed-form reward-reweighted policy, then applies its exponential reweighting to the observed response to form a practical target.For negative feedback, the target reduces the probability of the specific erroneous response.Source: §3.2 · p. 5

3.3 EFFICIENT PARAMETER UPDATE VIA LINEARIZED OPTIMIZATION

  • A first-order Taylor approximation turns target matching into a linear system, and Conjugate Gradient solves it without materializing the Hessian-like matrix.The matrix-free computation uses Jacobian-vector and vector-Jacobian products, reducing memory overhead.Source: §3.3 · p. 6

4 THEORETICAL RESULTS

The theoretical analysis establishes productive corrective updates, bounds their accumulated gains, and then accounts for approximation error in the practical update. The guarantees concern reliable negative feedback and balance progress toward acceptable responses against update error.

4 THEORETICAL RESULTS

  • The analysis proceeds from per-step productivity to cumulative gains, then bounds the practical update while accounting for approximation error.Its three stages address corrective steps, accumulated progress, and errors introduced by the efficient update.Source: §4 · p. 6

4.1 MONOTONIC ERROR REDUCTION

  • Each exact corrective update after reliable negative feedback is guaranteed to reduce KL divergence from the user policy to ROSA’s target policy.The reduction is proportional to the model’s prior probability of the rejected response, with β controlling its scale.Source: §4.1 · pp. 6–7

4.2 CUMULATIVE CONVERGENCE GUARANTEE

  • Across K exact corrective updates, cumulative progress toward acceptable responses is bounded, but convergence to a particular acceptable-response distribution requires additional conditions.The result assumes reliable rejected-response updates and finite initial KL divergence.Source: §4 · p. 6, §4.2 · p. 7

4.3 UNIFIED ERROR BOUND FOR THE ADAPTED POLICY

  • With log πθ Lipschitz-smooth, ROSA’s unified bound balances feedback-driven gains against first-order approximation error; β trades faster progress against update stability.Smaller β can accelerate learning while increasing approximation risk; larger β stabilizes updates but slows progress.Source: §4.3 · pp. 7–8

5 EMPIRICAL RESULTS

ROSA outperforms standard multi-turn baselines across diverse task domains and aligns responses with varied user preferences. It also supports online correction over extended interactions while maintaining general knowledge and requiring negligible GPU memory overhead.

5.1 EFFECTIVENESS AND GENERALIZABILITY ACROSS TASK DOMAINS

  • ROSA consistently outperforms standard multi-turn baselines across mathematical, general, code, and multilingual domains, matching or exceeding costly training and advanced test-time compute baselines.The reported comparisons also describe scaling to massive foundation models.Source: §5.1 · p. 9
  • ALOE mean Alignment Level rises from 3.78 to 4.40, IFEval strict accuracy from 58.31% to 70.26%, and MT-Bench subjective alignment from 60.36 to 75.57.The paper also reports alignment in delayed-reward UI Agent scenarios.Source: §5.1 · p. 9

5.2 EFFECTIVENESS IN ONLINE ERROR CORRECTION

  • ROSA sustains more newly solved problems deeper into conversation than zero-shot baselines or direct RL, whose correction curves plateau or improve slowly.The paper relates this online correction pattern to progressively learning from consecutive failures.Source: §5.2 · p. 9
  • ROSA remains stable for up to 20 turns, preserves general knowledge without catastrophic forgetting, and incurs negligible GPU memory overhead.It also overtakes static baselines in time-to-accuracy efficiency.Source: §5.2 · p. 10

6 CONCLUSIONS AND LIMITATIONS

The paper presents ROSA as an efficient way to adapt LLM policies during multi-turn interactions, while noting weaker performance when tasks depend heavily on pretrained knowledge.

6 CONCLUSIONS AND LIMITATIONS

  • ROSA enables efficient in-conversation self-correction through real-time parameter updates, but is less effective on tasks heavily bottlenecked by pretrained knowledge.Source: §6 · p. 10

AI USE STATEMENT

  • The authors report using generative AI to create and refine figures and copy-edit prose, with authors verifying figures and reviewing edits for technical changes.Source: AI USE STATEMENT · p. 11

A RELATED WORK

  • Related work positions ROSA against context-based prompting, offline multi-turn learning, and sampling-based test-time methods, emphasizing feedback-driven online updates with no additional offline training cost.Source: A RELATED WORK · p. 18

B.1 PROOF OF THEOREM 1

  • The distribution-space optimum has zero KL divergence over distributions supported on the reference policy, but a restricted parameter update need not realize that optimum.Theorem 4 accounts for this approximation.Source: §B.1 · p. 19

B.2 DERIVATION OF EQUATION 3

  • Under the single-sample feedback constraint, Equation 3 builds a practical target from the observed response’s reward and a surrogate for unobserved responses, without assuming their true utility is zero.Source: §B.2 · p. 19, §1 · p. 19

B.3 PROOF OF THEOREM 2

  • With exact updates and reliable rejection of unacceptable outputs, the stated guarantee applies; a noisy negative score on an acceptable output does not meet its premise.Source: §B.3 · p. 20

B.4 PROOF OF THEOREM 3

The analysis bounds improvement in unacceptable-answer probability and characterizes when approximate updates can make progress. Its finite-run sample bound is conditional on a sequence of reliable rejections.

B.4 PROOF OF THEOREM 3

  • Conditionally on K reliable rejections with p_k-1(y_k)≥p_min, K≥⌈(D0−η)/(Cp_min)⌉ suffices for D_K≤η; this is not an unconditional sampling-complexity guarantee.The bound assumes D0=DKL(P∥p0)<∞ and 0<η<D0.Source: §B.4 · p. 20
  • Rejection reduces unacceptable-answer probability but does not identify an arbitrary preferred distribution among acceptable answers; without further distinguishing feedback, their relative probabilities remain as initially conditioned on acceptability.Source: §B.4 · p. 21

B.5 PROOF OF THEOREM 4

  • Approximate updates require positive net progress, C p_k-1(y_k)>ϵ_k+L∥∆θ_k∥^2/2; CG residual reduction alone does not establish this condition at every update.Target realizability and appropriate conditioning also matter for small fitting error.Source: §B.5 · p. 22

C EXPERIMENTAL SETTING

  • The evaluation covers ROSA across diverse tasks and models to assess generalizability, effectiveness, and efficiency.Source: C EXPERIMENTAL SETTING · p. 22

C.1 DATASETS.

The evaluation spans multiple benchmark domains, model scales, feedback signals, and update mechanisms, then tests protocol consistency, feedback-template robustness, and time-to-accuracy.

C.1 DATASETS.

  • Four benchmark domains cover mathematics, general reasoning, code generation, and multilingual reasoning; final evaluations use held-out tests except specified training-set analyses.SuperGPQA training data is sampled for testing, and training or development sets are used when no dedicated test set exists or for ablations.Source: §C.1 · pp. 22–23

C.2 MODELS

  • The evaluation includes compact, larger, and reasoning-focused instruction-tuned models, with controlled diagnostics separated from cross-model experiments.Most controlled MATH diagnostics use Qwen3-0.6B; ALOE and successful-trajectory experiments use Qwen3-8B and are reported separately.Source: §C.2 · p. 23

C.3 EVALUATION METRICS

  • Performance is measured by accuracy within K turns and correction uplift after first-turn failures, while efficiency is measured by generation-update latency and peak GPU memory.Accuracy counts unique problems solved within the turn budget.Source: §C.3 · p. 24

C.4 REWARD MODELS

  • The study uses dense Qwen3-30B-A3B scores in [−1.0,+1.0] and sparse rule-based rewards of {−1,+1}; GPT-5.6 rubric calibration yielded Fleiss’ κ=0.920.Five annotators rated 50 score-stratified MATH responses; the calibration does not establish populationwide reliability or calibrate the separate ALOE judge.Source: §C.4 · pp. 24–25

C.5 PARAMETER UPDATE MECHANISMS

  • ROSA updates either small LoRA matrices on the LM head or the final hidden-state activation, leaving persistent model weights unchanged in the latter mode.The LoRA update confines optimization to a small parameter subset; hidden-state modification is transient and state-dependent.Source: §C.5 · pp. 25–26

C.6 MULTI-TURN EVALUATION PROTOCOL

  • The primary verifiable-task protocol allows up to 10 turns, stops after verified success, resets adaptation between problems, and gives baselines and ROSA identical initial prompts and simulated feedback.The scalar reward used for adaptation is specified separately from the shared textual-feedback construction.Source: §C.6 · p. 26

C.7 ROBUSTNESS TO FEEDBACK TEMPLATES

  • On MATH with Qwen3-0.6B, ROSA outperforms the baseline across tested feedback templates, with relatively small accuracy variation, especially for the model-based reward variant.The tested templates represent default, direct, and guiding feedback formulations.Source: §C.7 · p. 27, Table 8 · p. 27

D.1 TIME-TO-ACCURACY EFFICIENCY

  • ROSA’s accuracy initially lags from update cost, but its steeper learning curve later overtakes the baseline within equal wall-clock time.The authors attribute the crossover to more effective error correction despite additional per-turn computation.Source: §D.1 · p. 27

D.2 MEMORY AND LATENCY OVERHEAD

ROSA adds latency for online updates, while its longer-run time-to-accuracy advantage and modest memory increase characterize the overhead trade-off.

D.2 MEMORY AND LATENCY OVERHEAD

  • The average update time makes ROSA’s total processing time per turn roughly double the baseline’s inference-only time.The authors describe asynchronous execution as a way to perform updates in the background while users interpret responses and formulate their next prompt.Source: §D.2 · pp. 27–28
  • ROSA’s learning curves are steeper across the compared models and datasets, with accuracy initially lower at equal elapsed time before later overtaking the baseline.The initial lag is attributed to gradient calculation and backpropagation; the later crossover reflects more effective error correction.Source: Figure 5 · p. 28
  • +0.7 GB is the maximum memory increase reported for ROSA on Qwen3-32B; the matrix-free Conjugate Gradient solver avoids materializing full Hessian-like matrices.For models up to 8B parameters, the update peak shows only a modest increase over the inference peak.Source: §D.2 · p. 28, Table 10 · p. 28, Table 9 · p. 28

E.1 GENERALIZATION TO OPEN-ENDED AND DELAYED-REWARD TASKS

ROSA is evaluated on delayed-reward UI-agent tasks and subjective or open-ended tasks, then compared with test-time compute baselines on MATH.

E.1 GENERALIZATION TO OPEN-ENDED AND DELAYED-REWARD TASKS

  • On OSWorld and AndroidWorld, ROSA yielded consistent success-rate improvements after UI-TARS-7B agents received a single binary reward following trajectories averaging around 15 turns.For failed tasks, ROSA applies one parameter update based on the entire trajectory.Source: §E.1 · p. 29, Table 11 · p. 29
  • On MT-Bench, ROSA improved the subjective score metric by +15.21, and on IFEval, strict accuracy increased by +11.95.MT-Bench uses continuous GPT-5.2 judge rewards, while IFEval uses a programmatic structural-constraint checker.Source: §E.1 · p. 29, Table 12 · p. 29

E.2 COMPARISON WITH TEST-TIME COMPUTE BASELINES

  • Against test-time compute baselines on MATH with Qwen3-0.6B, ROSA achieved the highest accuracy and correction uplift with competitive time overhead.The comparison limits each method to at most 10 interactions or samples per problem.Source: §E.2 · p. 29, Table 13 · p. 30

T 2PAM (Ours) ROSA (+LM+R) 50.20 48.87 ∼1.8x

Across the reported evaluations, ROSA improves accuracy and error correction over baselines while using test-time policy adaptation. The supplementary studies examine its scalability, stability, reward types, and comparisons with other adaptation strategies.

T 2PAM (Ours) ROSA (+LM+R) 50.20 48.87 ∼1.8x

  • ROSA significantly outperforms compared baselines in final accuracy and correction, while targeted updates use a fraction of search-and-sampling computation.Search and sampling incur approximately 10x latency, compared with approximately 1.8x for ROSA.Source: T 2PAM (Ours) ROSA (+LM+R) 50.20 48.87… · p. 30

E.3 SCALABILITY TO LARGE FOUNDATION MODELS

  • On Qwen3-32B, ROSA raises MATH accuracy by +11.40% to 87.80%, with gains of +12.42% on MMLU-R and +9.34% on MT-AIME24.Peak GPU memory overhead is +0.7 GB.Source: §E.3 · p. 31, Table 14 · p. 31

E.4 STABILITY IN EXTENDED INTERACTION HORIZONS

  • At turn 20, ROSA reaches 64.80% accuracy while the in-context baseline stagnates after turn 15; ROSA also continues correcting problems through turns 5–10.The extended-horizon results do not establish that the conditions of the positive-net-progress regime hold at every empirical update.Source: §E.4 · p. 31, Table 15 · p. 31, §E.10 · p. 35

E.5 INTEGRATION WITH PROCESS REWARD MODELS (PRM)

  • With process-reward feedback, ROSA-PRM reaches 60.40% final accuracy and reduces average turns to solve from 6.9 to 5.8.The update targets the flawed reasoning-step prefix identified by the process reward model.Source: §E.5 · p. 32, Table 16 · p. 32

E.6 GENERAL KNOWLEDGE RETENTION

  • After 10 consecutive ROSA updates, MMLU accuracy and Wikitext perplexity remain practically intact, with average KL divergence from the base model of 0.001.The stress test applies updates to unresolved MATH problems before evaluating knowledge retention and calibration.Source: §E.6 · pp. 32–33, Table 17 · p. 33

E.9 COMPARISON WITH MULTI-TURN TRAINING METHODS

  • On MATH, ROSA is comparable to or better than multi-turn SFT and RL, without additional offline training cost.Additional benchmark results also report improved overall task performance and self-correction.Source: §E.9 · p. 34, §E.8 · p. 34, Table 19 · p. 34

E.12 ADAPTING TO INDIVIDUAL PREFERENCES ON ALOE

  • On ALOE, ROSA reaches mean alignment 4.99 at turn 10 versus 4.70 for the baseline; multi-sample feedback yields 52.60% accuracy at approximately 6.9x baseline latency.The ALOE comparison uses model-judged persona feedback; the multi-sample variant solves in 6.5 turns versus 6.7 for standard ROSA.Source: §E.7 · p. 33, Table 18 · p. 33, §E.12 · p. 36

F.1 THE IMPORTANCE OF THE OPTIMIZATION STRATEGY

  • On MATH with Qwen3-0.6B, ROSA exceeds the direct RL baseline by +24.00% accuracy and +31.31% Correction Uplift on MATH-500.The ablation attributes the gains to deriving a stable target policy and solving for the parameter update, rather than merely adding an online reward signal.Source: §F.1 · p. 37, Table 24 · p. 37, Table 25 · p. 37

F.2 ABLATION STUDY ON THE INFLUENCE OF HYPERPARAMETER β

The ablation varies β to examine its effect on ROSA’s update intensity and performance. Smaller β produces more aggressive updates, but the tested settings reach similar final accuracy.

F.2 ABLATION STUDY ON THE INFLUENCE OF HYPERPARAMETER β

  • β controls update intensity: smaller values amplify the reward signal for more aggressive updates, while larger values dampen it.The study concludes that final performance is not highly sensitive to β across the tested operational range.Source: §F.2 · p. 38
  • Across β values from 0.25 to 1.75, cumulative accuracy trajectories differ initially but converge to similar final accuracy over 10 turns.Figure 8 presents the cumulative accuracy curves for the tested values.Source: §F.2 · p. 38, Figure 8 · p. 38

G CASE STUDY

The case study follows the baseline and ROSA as they respond to feedback on converting (0,3) from rectangular to polar coordinates. Both responses ultimately give the stated correct answer.

G CASE STUDY

  • The cited baseline response and ROSA response both give (3, pi/2), matching the stated true answer (3, pi/2) for the point (0, 3).Source: G CASE STUDY · p. 39
Loading 2509.23166v3…