PaperlayerView exact revision on arXiv

Source-linked AI summary

Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation

Chenxing Wei, Hong Wang, Ying He, Zhongxiang Dai, Bo Jiang, F. Richard Yu, Yao Shu

arXiv:2603.01375v1cs.AIcs.LG

TL;DR

Test-time policy adaptation must address both ambiguous context and model capability, whereas existing paradigms commonly optimize only one axis. ROSA2 jointly adapts Words and Weights using textual gradients and parameter updates, with reported gains in accuracy and interaction efficiency. The paper also frames its convergence claims under explicit smoothness and scaling assumptions.

  • Problem

    Existing test-time adaptation paradigms commonly refine instructions or adjust weights alone, despite T2PAM involving both context ambiguity and model capability.

  • Method

    ROSA2 reformulates T2PAM as joint optimization over feedback context and model parameters, approximating the interaction objective’s full gradient through co-adaptation.

  • Results

    ROSA2 reports a 30% average accuracy improvement across multiple benchmarks while reducing average interaction turns by 40%.

  • Takeaways & Limitations

    The reported results support co-adapting semantic context and parameters to bypass the local minima associated with conditional baselines.

  • Takeaways & Limitations

    The unified convergence bound assumes that log π(y | x, θ) is L-Lipschitz smooth and treats Z_t as constant with respect to current optimization variables.

Abstract

from arXiv · show

Test-time policy adaptation for multi-turn interactions (T2PAM) is essential for aligning Large Language Models (LLMs) with dynamic user needs during inference time. However, existing paradigms commonly treat test-time adaptation as a single-axis problem, either purely refining instructions (Prompt Engineering) or only adjusting weights (Test-Time Training), ignoring that interaction failures stem from a coupled mix of ambiguity and incapacity. We argue that these two optimization paths are not merely additive but synergistic: semantic clarity acts as a pre-conditioner for effective parameter updates. To this end, we propose ROSA2, a framework that reformulates interaction as a joint optimization problem over the heterogeneous space of Words and Weights. By mathematically decomposing the error signal, ROSA2 utilizes textual gradients to rectify intent ambiguity and parameter updates to bridge capability gaps. Theoretically, we prove that this co-adaptation strictly reduces the required parameter shift for convergence. Empirically, ROSA2 outperforms state-of-the-art baselines by 30% on MATH while reducing interaction turns by 40%, demonstrating that refining the context unlocks the true potential of parameter updates.

1. Introduction

ROSA2 reframes test-time policy adaptation as joint optimization over semantic context and model parameters, addressing ambiguity and capability gaps together. The framework is theoretically motivated and empirically reported to improve accuracy while reducing interaction turns.

  • T2PAM adapts model policies during multi-turn sessions to align responses with specific user preferences.
  • Existing approaches optimize either instructions or model weights, leaving the coupled sources of interaction error unresolved.
  • ROSA2 jointly adapts semantic context and model parameters by using textual gradients to clarify intent and parameter updates to improve execution.
  • +30.8% on MATH is reported for ROSA2 alongside nearly 40% fewer interaction turns.The contribution summary identifies these as evaluations across diverse domains and reports negligible memory overhead.
  • The framework’s optimization landscape contrasts a Deficit Trap for Prompt-Only methods with an Overfitting Trap for Param-Only methods.

2. Motivation: The Traps of Conditional Optimization

T2PAM requires jointly addressing context ambiguity and model capability, because optimizing prompts or parameters alone can become trapped in different failure modes. ROSA2 co-adapts both dimensions to sustain improvement.

  • The joint optimization challenge: T2PAM jointly optimizes context x (Words) and model weights θ (Weights), whereas single-axis updates can leave reasoning deficits or overfit noisy prompts.The conditional strategies are hypothesized to converge to suboptimal states when either context or parameters remain fixed.
  • Experimental setup: Controlled experiments on Qwen3-8B using MATH simulate a challenging 10-turn interaction and compare optimization settings that isolate prompt and parameter updates.The study includes standard inference and conditional optimization settings, with additional switch experiments changing the optimized dimension.
  • Empirical failure modes: Prompt-Only and Param-Only methods show diminishing returns and premature stagnation: semantic updates fail to bridge reasoning gaps, while parameter updates plateau through overfitting.The frozen-parameter Prompt-Only method is associated with policy misalignment, whereas Param-Only lacks context refinement.
  • Theoretical interpretation: The Deficit Trap arises when frozen parameters prevent semantic updates from correcting intrinsic reasoning deficits, while the Overfitting Trap arises when weights adapt to ambiguous prompts without context refinement.These traps map the empirical stagnation of Prompt-Only and Param-Only optimization to distinct regions of the theoretical landscape.
  • Escaping conditional traps: Switch experiments escape the local minima after introducing the missing optimization dimension, supporting the need for both semantic clarity and parametric capability.The experiments use the added optimization dimension to move beyond the conditional-update traps.
  • ROSA2: ROSA2 jointly optimizes words and weights from the first turn by approximating the full interaction-objective gradient, aiming to bypass both traps along an Optimal Trajectory.Figure 2(a) presents ROSA2 as the Red Solid trajectory with faster convergence.

3. Joint Optimization via Full-Gradient Approximation

ROSA2 formulates T2PAM as joint optimization over context and model parameters, using a full-gradient view to align the policy with a joint optimal policy. Its protocol alternates generation and evaluation with semantic and parametric updates across turns.

  • Problem formulation: ROSA2 models the policy as jointly dependent on Words context and Weights parameters, optimizing both rather than either variable alone.The joint variables are the next-turn query x_t+1 and adapter weights θ_t.
  • Joint optimal policy construction: The framework constructs a joint optimal policy by reweighting the previous-turn policy with reward feedback, while treating the partition function as constant during the current update.The target π*_t is built from π_t−1, and Z_t is constant with respect to ϕ_t = {x_t+1, θ_t}.
  • Optimization objective: ROSA2 minimizes forward KL divergence to the joint optimal policy, which is equivalent to maximizing its expected log-likelihood when the target entropy is fixed.This equivalence follows from expanding the KL divergence and treating the target-policy entropy as independent of the optimizable variables.
  • Full-gradient co-adaptation: The full-gradient update simultaneously rectifies the current query and updates parameters along a reward-weighted log-likelihood direction.The method uses semantic gradients for context refinement and parameter gradients for adapter updates.
  • ROSA2 protocol: Each rejected turn triggers semantic refinement of feedback and a parametric update, after which the refined query and response are added to the interaction history.If explicit feedback is absent, the semantic stream can synthesize a corrective query from the current failure.
  • ROSA2 protocol: The framework is presented as addressing ambiguity and capability together: the Semantic Stream clarifies feedback, while the Parametric Stream improves model execution.This co-adaptation is intended to provide clear instructions and stronger execution in subsequent turns.

4. Theoretical Results

ROSA2’s theoretical analysis shows that jointly refining context and model parameters reduces the parameter shift needed for alignment and yields a unified convergence bound. Figure 3 supports these results by showing lower parametric and total errors despite semantic costs.

  • 4. Theoretical Results: ROSA2 jointly updates query context x and parameters θ to drive the model policy toward the latent optimal user policy.The analysis treats joint updates as the basis for convergence toward the user-optimal policy.
  • 4.1. Mechanism: Parametric Error Reduction: Semantic refinement strictly reduces the norm of the parameter shift required to reach alignment.Theorem 4.1 frames context refinement as a pre-conditioning step that minimizes residual error for parameter updates.
  • 4.2. Unified Convergence Bound: Theorem 4.2 decomposes convergence into initial error, feedback-driven improvement, and approximation error after T turns.The bound accounts for semantic and parametric update steps and quantifies divergence from the optimal policy.
  • 4.1. Mechanism: Parametric Error Reduction: Figure 3 shows ROSA2’s parametric error significantly below the baseline, supporting Theorem 4.1.The figure compares baseline parametric error with ROSA2’s decomposed errors.
  • 4.2. Unified Convergence Bound: ROSA2’s total error remains below the baseline despite an additional semantic cost, whose approximation component decays exponentially.The total error initially rises with semantic discrepancy but rapidly drops below the single-stream baseline.

5. Empirical Results

Across reasoning, agentic, and deployment evaluations, ROSA2 outperforms single-axis methods while improving correction efficiency and maintaining manageable computational costs.

  • 5.1. Performance and Efficiency in Diverse Tasks: ROSA2 consistently outperforms TextGrad and ROSA across model sizes from 0.5B to 8B and across evaluated domains.TextGrad faces capability ceilings, whereas ROSA can overfit ambiguous inputs; ROSA2 addresses both bottlenecks.
  • 5.1. Performance and Efficiency in Diverse Tasks: 81.4% Correction Uplift on MATH shows ROSA2’s ability to correct problems after initial failure.Correction Uplift measures the percentage of eventually solved problems corrected after the initial failure.
  • 5.1. Performance and Efficiency in Diverse Tasks: -40% Avg Turn compared to ROSA shows that ROSA2 reaches solutions in fewer interaction turns.The reduction in turns provides empirical support for the convergence analysis.
  • 5.2. Adaptability in Sparse-Reward Environments: ROSA2 improves UI-agent adaptability across OSWorld and AndroidWorld with both SFT and DPO backbones.The method uses retrospective semantic feedback to convert sparse failures into more detailed supervision for parameter updates.
  • 5.3. Computational Cost Analysis: ROSA2 reduces average time per problem through shorter reasoning trajectories and fewer total conversation turns.Its memory overhead reaches a maximum of +3.1 GB on MATH.

6. Conclusions

ROSA2 jointly optimizes context and parameters to resolve error attribution, achieving state-of-the-art accuracy with reduced latency across diverse benchmarks.

  • 6. Conclusions: ROSA2 jointly optimizes context and parameters, bypassing local minima associated with conditional baselines.The conclusion reports state-of-the-art accuracy with reduced latency across diverse benchmarks.

Impact Statement

The paper positions ROSA2 as a contribution to test-time adaptation for multi-turn interactions, emphasizing technical efficiency, accuracy, and more capable UI agents.

  • Impact Statement: ROSA2’s co-adaptation of context and parameters is presented as advancing test-time adaptation for multi-turn interactions.The stated scope includes technical efficiency and accuracy in LLMs.
  • Impact Statement: The framework is reported to achieve state-of-the-art performance on reasoning and agentic benchmarks.The impact statement connects this performance to the potential for more capable UI agents.
  • Impact Statement: Prompt Engineering and Test-Time Training are described as the two single-axis adaptation paradigms motivating the joint approach.The related-work passages distinguish context refinement from parameter updates.

B.1. Proof of Theorem 4.1

The proof shows that refining semantic context reduces the residual driving parameter updates, thereby shrinking the required parameter shift and supporting convergence toward the optimal policy.

  • Residual-Driven Update: ROSA’s parameter update solves a least-squares residual-fitting problem between the target distribution and current policy.The update is governed by (J^T J)Δθ = J^T R(x), with R(x) defined as the target-minus-current policy residual.
  • Conclusion: A smaller residual bounds the parameter update by a smaller magnitude, so optimizing the query reduces the required parameter shift.The conclusion follows by substituting the reduced residual into the update bound.
  • Effect of Semantic Refinement: Semantic refinement brings the current policy closer to the user’s optimal policy and reduces the residual vector.The Semantic Stream updates x_t toward x* to minimize semantic discrepancy, yielding a smaller residual after refinement.
  • Global Convergence: The global convergence analysis decomposes total error into an ideal gain and an approximation cost.It uses a theoretical target, monotonic error reduction, an L-Lipschitz smoothness assumption, and telescoping over T turns to derive Theorem 4.2.
  • Global Convergence: Summing the per-turn bounds yields the unified convergence bound of Theorem 4.2.The bound accounts for approximation errors while quantifying divergence reduction from the optimal policy.

C. Experimental Setup

The experiments evaluate ROSA2’s efficacy, efficiency, and generalizability across varied tasks and model architectures using specified datasets, metrics, and reward mechanisms.

  • Experimental Scope: The evaluation spans diverse tasks and model architectures to assess ROSA2’s efficacy, efficiency, and generalizability.The experimental section specifies datasets, models, evaluation metrics, and reward mechanisms.

C.1. Datasets

The study evaluates ROSA2 across mathematical, general, code-generation, and multilingual reasoning benchmarks, using models that vary in scale, architecture, and reasoning specialization.

  • Domains: The benchmark suite covers four domains: Mathematical Reasoning, General Reasoning, Code Generation, and Multilingual Reasoning.Table 5 summarizes the dataset statistics.
  • Mathematical Reasoning: Mathematical Reasoning uses MATH, AIME25, and MATH-500 for complex multi-step problem solving.These benchmarks range from 12,500 high-school competition problems to a 500-problem evaluation subset and 25 difficult AIME problems.
  • General Reasoning: General Reasoning uses GPQA-diamond, MMLU-Redux, and SuperGPQA to test expert-level knowledge application.The datasets cover graduate-level questions, 57 subjects, and 285 disciplines.
  • Code Generation: Code synthesis is evaluated with HumanEval, which contains 164 programming problems verified by function signatures, docstrings, and unit tests.Functional correctness is checked through the associated tests.
  • Multilingual Reasoning: Multilingual reasoning is evaluated on translated IMO, AIME, and MATH problems through M-IMO, MT-AIME24, and MT-MATH100.The evaluation uses the multilingual versions provided by MCLM.
  • Evaluation Protocol: Evaluation primarily uses official held-out test sets, with specified training-data sampling only where standard test sets are unavailable or for ablations.SuperGPQA is the stated case where a portion of training data was sampled for testing.
  • Model Selection: The model set includes compact and larger Qwen variants plus reasoning-focused DeepSeek-R1 distilled models.Qwen3 and DeepSeek-R1 models also provide validation against potential Qwen2.5 benchmark contamination.

C.3. Evaluation Metrics

The evaluation measures downstream performance and computational efficiency through turn-bounded accuracy, self-correction, time, memory, and reward mechanisms with different feedback granularity.

  • Evaluation Dimensions: The framework evaluates downstream task performance and computational efficiency.These are the two stated critical dimensions of evaluation.
  • Performance Metrics: Accuracy is the proportion of unique problems solved within a maximum of K conversational turns.Each problem contributes an indicator of whether it is solved at any turn t ≤ K.
  • Performance Metrics: Correction Uplift measures the percentage of problems initially failed at turn 1 that are solved in later turns.It quantifies the model’s capacity to self-correct.
  • Efficiency Metrics: Efficiency tracking includes average time to solve each problem and peak GPU memory during inference and updates.Peak GPU Memory is the maximum observed VRAM usage.
  • Reward Mechanisms: Two reward mechanisms simulate different feedback granularities, including a sparse rule-based model that assigns +1.0 for exact matches and −1.0 otherwise.The rule-based model extracts a final answer and compares it with ground truth.
Loading 2603.01375v1…