RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation

Motivation

• Terminal scores indicate whether an interaction succeeded but do not localize which behaviors caused failure across interdependent decisions.In long-horizon settings, each action changes the partner’s subsequent behavior, and final outcomes emerge across many turns.Source: §1 · p. 1
• Response frequency or ease of evaluation need not identify recurrent causes of task failure, so reward adaptation should be driven by recurrent failures in low-outcome trajectories.Figure 1 contrasts fixed Rubrics, external revision, joint co-evolution, and RewardWeaver’s prioritization of current bottlenecks.Source: §1 · p. 2, Figure 1 · p. 2
• Capability bottlenecks for training credit must be outcome-grounded, recurrent across failed trajectories, and attributable to the agent rather than its partner or scenario.The task-level objective remains stable while process-level credit shifts toward capabilities that currently constrain performance.Source: §1 · p. 2

RewardWeaver

• RewardWeaver identifies recurrent capability bottlenecks from on-policy failures and selects corresponding Rubrics for the next process reward, keeping admitted semantics fixed and validating uncovered failures separately.Its loop connects policy training, automatic evaluation, failure attribution, and reward adaptation.Source: §1 · p. 2, Figure 2 · p. 3

Evaluation and contributions

• RewardWeaver achieves new state-of-the-art performance under matched protocols, and its analyses identify failure-grounded attribution, adaptive credit reallocation, and stable admitted-capability semantics as contributors to gains.The evaluation covers SOTOPIA and related social intelligence benchmarks.Source: §1 · pp. 2–3

2 RewardWeaver Methodology

• The method separates stable reward semantics from stage-dependent relevance: a validated capability space defines what can be rewarded, while an outer loop selects what receives credit.The capability library expands only through a separate validated procedure.Source: §2 · p. 3

2.1 Task Formulation

• Each trajectory receives a fixed task-level outcome and a process reward formed from the active capability Rubrics, whose scores contribute equally.The active set determines which capabilities receive credit, rather than how much each receives.Source: §2.1 · p. 3

Algorithm 1 Validated Capability-Space Construction

• Capability-space construction induces candidates from interaction evidence, consolidates and filters them, then operationalizes and tests candidates before admission.Candidates are filtered for generality, operationality, and semantic distinctiveness; validation can lead to revision or rejection.Source: §2.2 · p. 4, Algorithm 1 Validated Capability-Space… · p. 4

2.2 Validated Capability Space

• Admitted capabilities pair observable, policy-controllable behaviors with frozen Rubric definitions and evaluators that pass source-disjoint acceptance testing.Candidate discovery remains flexible, but admitted capability semantics remain fixed.Source: §2.2 · p. 4

2.3 Outcome-Grounded Failure Attribution

• After each stage, low-outcome on-policy trajectories are diagnosed, candidate capabilities are retrieved, and unsupported assignments are rejected.Aggregated records retain recurrent, supported, policy-controlled failure modes rather than isolated low scores.Source: §2.3 · p. 4, Algorithm 2 Failure Attribution and… · p. 5

Algorithm 2 Failure Attribution and Adaptive Credit Allocation

• The attribution-and-allocation update aggregates records, scores eligible capabilities, selects a budgeted active set, and calibrates the next reward before publication.If calibration fails, the algorithm returns failure and publishes no new reward.Source: Algorithm 2 Failure Attribution and… · p. 5

2.4 Adaptive Credit Allocation

• All active Rubrics contribute equally, and a receding bottleneck can leave the active set so capacity shifts to newly dominant failures.The update changes active Rubrics and their composition, not the frozen semantics or evaluators of admitted capabilities.Source: §2.4 · p. 5

2.5 Controlled Capability Expansion

• When recurrent failures fall outside the current capability space, validated additions become eligible only in a subsequent reward update.Training priorities evolve rapidly, while reward semantics evolve conservatively.Source: §2.5 · pp. 5–6

3.2 Experimental Results and Analysis

• On SOTOPIA-Hard, Goal/Overall rises from 7.67/3.94 to 7.90/4.38 Self-Play and 7.30/3.80 to 7.66/4.22 Partner; Sales results lead on six of eight metrics.The evaluation spans SOTOPIA, AmazonHistoryPrice, and a 400-task Sales Benchmark, using Self-Play and Partner protocols where specified.Source: §3.2 · pp. 6–7, §3.1 · p. 6
• Under the DeepSeek-V4-Pro buyer, REWARDWEAVER reaches a 95.2% MI Deal Rate and the highest Normalized Seller Utility (0.9737).The figure places it on the upper-right frontier among evaluated methods.Source: §3.2 · p. 6, Figure 3 · p. 7
• Outcome-only GRPO and Static Top-K underperform REWARDWEAVER, while Random Active Set shows that changing rewards alone is insufficient without bottleneck alignment.The ablation comparisons use matched training budgets.Source: §3.2 · p. 7, Table 2 · p. 7
• Failure-frequency selection and random-trajectory attribution underperform REWARDWEAVER, while Success–Failure Trajectory Attribution remains competitive but does not consistently improve on it.The comparisons suggest that deficiency prevalence or outcome-agnostic behavior analysis alone does not reliably identify failure-relevant capabilities.Source: §3.2 · pp. 7–8
• Open-Vocabulary Dimensions is consistently weaker across four settings, while Instance-Specific Rubrics varies despite sometimes matching or exceeding individual metrics.The comparison supports maintaining stable, validated definitions for admitted capabilities.Source: Table 2 · p. 7, §3.2 · p. 8

3.3 Reward Adaptation Analysis

• Maintain Goal Commitment dominates early and later recedes as Switch Strategies, State Benefits, and Respond with Empathy gain prominence; Conditional Concessions re-emerges after declining.The priority shifts indicate stage-wise reallocation rather than cumulative addition of reward Rubrics.Source: §3.3 · p. 8, Figure 4 · p. 8
• Bottleneck importance positively correlates with next-stage failure reduction (Spearman ρ = 0.28), versus ρ = 0.21 for current failure rate alone.The association remains positive after controlling for failure rate (partial ρ = 0.22) and strengthens within stage (ρ = 0.31); the analysis is not causal identification.Source: §3.3 · p. 8
• B = 6 and a 30-step interval perform strongly, while tracked capabilities show no systematic regression after deactivation.Across 36 post-deactivation observations, scores remain within 0.02 of or above final active scores, with mean change +0.10.Source: §3.3 · pp. 8–9

Outcome and Preference-Based Rewards

• Outcome and preference-based methods provide rewards or feedback, but reward specifications are often fixed or set before training; which capabilities a policy should optimize next remains open.The discussion contrasts RLVR and process verifiers (Shao et al., 2024; Le et al., 2022; Lightman et al., 2024; Lu et al., 2024; Zeng et al., 2025; Setlur et al., 2025; Zheng et al., 2025) with preference-based approaches.Source: §4 · p. 9

Rubric-Based Process Rewards

• Rubric methods improve process-quality evaluation, but discriminability or deficiency prevalence does not establish which criteria recurrent low-outcome failures implicate.The passage discusses instance-specific and reusable rubrics, multidimensional evaluation, checklist scoring, and response-distribution adaptation.Source: §4 · p. 9

Credit Assignment in Multi-Turn Interaction

• Prior credit-assignment methods locate feedback within an interaction, whereas reward adaptation selects what process-reward capabilities to include.The passage distinguishes this question from methods for delayed returns, utterance- or segment-level feedback, automated attribution, and dialogue planning.Source: §4 · p. 9

• Reward adaptation is an outer-loop problem: standard RL optimizes under a given reward, while the outer loop selects what should receive training credit next.Source: §5 · p. 9
• The reward priorities should track current performance bottlenecks rather than merely capabilities that are easy to evaluate.Source: §5 · p. 9
• LLM-based evaluator errors or uninformative terminal outcomes can propagate into reward adaptation.Source: §5 · p. 9
• The active budget and adaptation interval may require retuning across domains.Source: §5 · p. 9

AI Use Statement

• Generative AI supported methodology and experimental-design discussions, qualitative analysis, manuscript drafting and editing, and presentation of selected materials; authors reviewed the content and verified empirical details.Source: AI Use Statement · pp. 9–10

A Validated Capability-Space Construction for SOTOPIA in RewardWeaver

• The appendix presents the capability-space structure, representative Rubric entries, and an example of converting a frozen Rubric into an executable evaluator.Source: A Validated Capability-Space… · p. 13

A.1 Capability-Space Overview

• The SOTOPIA capability space contains 40 individual Rubrics across seven behavioral families, with adaptation operating on individual Rubrics rather than family-level scores.Source: Table A1 · p. 13, §A.1 · p. 13

A.2 Representative Capability Rubrics

• Representative entries span major capability areas, and their positive and negative evidence illustrates each Rubric’s intended semantic boundary.Source: §A.2 · p. 13, Table A2 · p. 14

A.3 Rubric-Specific Evaluator Operationalization

• For USE OF KEY INFORMATION, evaluator operationalization specifies applicability, scoring, and uncertainty handling while keeping the Rubric semantics fixed.Source: §A.3 · pp. 13–14, Table A3 · p. 13

B Sales Benchmark Construction and Evaluation

• The Sales Benchmark contains 400 health-service sales tasks across 20 scenario families, including 350 standard tasks and 50 constraint-conflict tasks.Source: B Sales Benchmark Construction and… · p. 15

B.1 Construction Protocol

• Tasks were authored with scenario specifications, buyer profiles, business rules, and role-specific goals and private information; GPT-5.6 Sol assisted drafting and refinement.Source: §B.1 · p. 15, §B.2 · p. 15
• The benchmark is entirely synthetic and targets interactive decision making and constraint following, not medical reasoning or real-world healthcare recommendations.Source: §B.1 · p. 15

B.2 Authoring Template and Quality Control

• Authoring and quality control preserve intentional goal conflicts while checking rule consistency, role visibility, observable completion conditions, and private-information separation.Source: §B.2 · p. 15

B.3 Scenario Families

• Table B1 summarizes the benchmark’s 20 scenario families, with abbreviated displayed names.Source: §B.3 · p. 15, B Sales Benchmark Construction and… · p. 15, Table B1 · p. 16

B.4 Split and Evaluation

• The fixed split assigns 320 tasks to training and 80 to evaluation; evaluation tasks share scenario families and profile banks but have no overlapping task IDs or serialized specifications.Source: §B.4 · p. 16
• Qwen2.5-72B scores outcomes with the seven-dimensional SOTOPIA-EVAL rubric, and Overall is the arithmetic mean of its seven raw dimension scores.Source: §B.4 · p. 16

B.5 Representative Tasks

• Representative tasks include eliciting buyer coverage and preferences before a replacement purchase, and honoring a free-transfer option after an explicit refusal of an upgrade.Source: §B.5 · pp. 16–17

C Additional Benchmark Results of RewardWeaver

• Tables C1 and C2 provide full AmazonHistoryPrice and Sales Benchmark results, respectively, but the supplied passages do not state their comparative values.Source: C Additional Benchmark Results of… · p. 18, Table C1 · p. 18, Table C2 · p. 18

D.1 Bottleneck Transition

• In a negotiation case, unclear communication accompanied Goal Completion 2/10; after Clear Expression entered the reward, sampled follow-ups scored 7 and 9 before emphasis shifted to Concrete Solutions.Source: §D.1 · p. 19
• Training priorities can change as bottlenecks recede and new failures become dominant, so process-level credit is stage-dependent rather than fixed.Source: §D.1 · p. 19

D.2 Contrastive Validation of Failure-Derived Rubrics

• A candidate Empathic Support evaluator passed all 23 applicable trajectories with little score variation, so it was flagged as collapsed and the rubric set failed its publication gate.Source: §D.2 · p. 19
• Failure evidence proposes candidate capabilities, but contrastive calibration tests whether generated Rubrics distinguish failure-relevant behavior and satisfy publication conditions.Source: §D.2 · p. 19

D.3 Selective Attribution under Uncertainty

• Low outcomes warrant diagnosis, but capability credit requires observable policy-controlled evidence, leaving partner-driven or ambiguous failures unattributed.A partner’s preference can explain a failed task without activating an agent-side capability.Source: §D.3 · pp. 19–20

E Association Between Bottleneck Attribution and Subsequent Failure Reduction

• The analysis asks whether estimated bottleneck importance is informative about subsequent changes in capability-level failure patterns.It examines this relationship across adjacent training stages.Source: E Association Between Bottleneck… · p. 21

E.1 Analysis Setup

• Across 240 capability–stage pairs, bottleneck importance contrasts low- and high-outcome attribution fractions, while the success-side fraction serves analysis rather than attribution.The outcome is the next-stage decrease in attributed failure rate; Spearman rank correlation evaluates the relationship because attribution prioritizes capabilities relatively.Source: §E.1 · p. 21

E.2 Results

• ρ = 0.22 (partial Spearman) after controlling for current failure prevalence, and ρ = 0.31 when comparing capabilities within the same training stage.Both associations remain positive.Source: §E.2 · p. 21
• ρ = 0.28 (Spearman) for bottleneck importance versus ρ = 0.21 for current failure rate alone, in association with next-stage failure reduction.The comparison measures capability-level signals at stage k against reductions in attributed failures at stage k + 1.Source: Table E1 · p. 21, §E.2 · p. 21
• Higher bottleneck-importance rankings are associated with larger subsequent reductions in capability-level failure rates, including beyond current failure prevalence.Source: §E.2 · p. 21

F.1 Active Rubric Budget

• B = 6 achieves the strongest results across reported settings under Self-Play and Partner evaluation, while performance is non-monotonic as the active Rubric budget changes.The comparison supports selective rather than cumulative allocation of training credit.Source: Table F1 · p. 22, §F.1 · p. 22

F.2 Adaptation Interval

• The 30-step interval yields the strongest Goal Completion scores under Self-Play and the strongest results across Partner settings, though 20 steps is marginally higher on selected Self-Play Overall scores.Performance is non-monotonic across adaptation intervals.Source: Table F2 · p. 22, §F.2 · p. 22

G.1 Evaluation Protocol

• On fixed diagnostic scenarios with a fixed evaluation criterion, retention tracks post-deactivation score changes from each capability’s final active stage, using continuously active families as references.The protocol includes six capabilities deactivated after the initial stage and one deactivated later.Source: §G.1 · p. 23

G.2 Retention Results

• Across 36 post-deactivation observations, mean change was +0.10, median +0.06, and minimum −0.01; none fell more than 0.02 below its final active score.Adapt to New Information remained at 1.00 throughout subsequent stages.Source: Table G1 · p. 23, §G.2 · p. 23
• After deactivation, Switch Strategies rose from 0.62 to 1.00, State Benefits from 0.84 to 0.98, and Address Concerns with New Options from 0.80 to 0.90.The later-deactivated Adapt to New Information was similarly stable.Source: §G.2 · p. 23
• On the fixed diagnostic set, shifting emphasis toward newly relevant capabilities did not systematically erase capabilities emphasized earlier.The authors characterize adaptive reward as reallocating training credit rather than requiring continual reinforcement of previously learned capabilities.Source: §G.2 · p. 23