PaperlayerView exact revision on arXiv

Source-linked AI summary

CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Wenhao Yu, Zhengcheng Wang, Jiahong Liu, Zexuan Qiu, Minda Hu, Shaohang Wei, Jiafei Lyu, Junyan Lyu, Haobo Fu, Irwin King

arXiv:2610.29965v1cs.LG

TL;DR

Joint RL and distillation provide complementary feedback, but opposing gradient components can reduce local optimization progress, and a mixing coefficient cannot selectively remove them. CORD combines the gradients by protecting one objective and projecting the other when the chosen condition is met. It improves all three evaluated ALFWorld recipes and two of three WebShop pairs, while its analyses identify configuration choices that remain consequential.

  • Problem

    Joint optimization of reward and distillation gradients can reduce first-order progress when their contributions conflict, while changing their mixing coefficient cannot selectively remove opposing components.

    Source: §1 · p. 2

  • Method

    CORD keeps one weighted objective gradient intact and minimally modifies the other according to a selected protection direction and projection gate.

    Source: §3 · p. 4

  • Results

    CORD improves all three host recipes on ALFWorld by 1.5–15.9 percentage points and two of three paired comparisons on WebShop.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Protection direction and triggering must be chosen together, while the distillation loss, mixing weight, and gradient aggregation remain consequential.

    Source: §4.3 · p. 10, §5 · p. 11

    The ideal combined-gradient properties do not necessarily describe the optimizer update direction, which can change under norm clipping, AdamW transformations, and weight decay.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their objectives. Across three such recipes, it improves ALFWorld success by 1.5-15.9 percentage points. On WebShop, it improves two of the three pairs, with SDAR rising from 60.7% to 64.6% success. Our analyses explain why protection direction and projection gate must be chosen together, and why the distillation objective and its mixing weight remain consequential after projection.

1 Introduction

Joint RL and distillation provide reward-based and dense teacher feedback, but their gradients can conflict, and a mixing coefficient cannot selectively remove opposing components. CORD projects one gradient against another, and the paper reports gains across joint-training recipes.

1 Introduction

  • The introduction situates the problem in sparse outcome feedback from GRPO and dense token-level teacher feedback from OPD.The usefulness of OPD depends on the teacher’s guidance.Source: §1 · p. 1
  • Joint optimization requires choosing which contribution to preserve because changing a mixing coefficient changes scale but cannot selectively remove opposing components.The introduction frames protection choice around the losses’ roles and training goal.Source: §1 · p. 2
  • 48.2% of mini-batch gradient records have negative RL–distillation cosine, and a conflicting distillation gradient reduces first-order progress on the RL surrogate relative to RL alone.The diagnostic uses the WebShop log described with Figure 1.Source: §1 · p. 2, Figure 1 · p. 2
  • On conflict, projecting one gradient onto the other’s orthogonal complement moves cosine quantiles to zero and recovers the protected gradient’s first-order progress; aligned pairs remain unchanged.CORD’s projection is applied selectively to the opposing component.Source: §1 · p. 2
  • CORD combines two objectives by retaining the selected protected gradient and minimally modifying the other before combination.The framework makes the projection trigger explicit; either objective can be protected.Source: §1 · p. 2
  • CORD improves all three host recipes on ALFWorld by 1.5–15.9 percentage points and two of three paired comparisons on WebShop.The paired comparisons retain each host recipe’s objectives.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe reward-based GRPO, teacher-guided on-policy distillation, and weighted-sum optimization. They define gradient conflict as opposing contributions that reduce first-order progress and motivate preserving one gradient while minimally modifying the other.

2.1 Group Relative Policy Optimization (GRPO)

  • GRPO compares returns across sampled attempts and assigns response records group-relative advantages for its clipped policy loss.The host specifies which response records enter the group statistics.Source: §2.1 · p. 3

2.2 On-Policy Distillation

  • On-policy distillation scores the student’s sampled tokens with a teacher rather than requiring teacher-generated trajectories.In the skill-conditioned setup, the teacher uses the current policy with additional skill context and disabled gradients.Source: §2.2 · p. 3

2.3 Multi-Objective Optimization and Gradient Conflict

  • For weighted losses F1 and F2, the combined gradient is u + v, where u = ∇F1 and v = λ∇F2; a negative cross term reduces each objective’s first-order progress relative to its own gradient.CORD chooses which weighted gradient to preserve and minimally modifies the other.Source: §2.3 · pp. 3–4
  • The agent illustration pairs GRPO’s environment rewards with OPD’s teacher guidance and shows projection removing the component of the second gradient that opposes the protected gradient.Either objective can be protected.Source: Figure 2 · p. 4

3 Method

CORD combines two objective gradients by explicitly choosing which objective to protect and when to project. It minimally changes the other gradient under a geometric constraint, with choices that affect how the mixing weight and accumulated mini-batch gradients shape the update.

3 Method

  • CORD keeps the selected weighted gradient intact and makes the smallest change to the other gradient that satisfies the chosen geometric constraint.Either objective can be protected; the losses and training goal inform that choice.Source: §3 · p. 4, §3.1 · p. 4

3.1 Choosing a protected direction

  • Swapping the protected and candidate gradients changes the combined direction without redefining the losses or fixing priority across problems.The protected loss is the corresponding weighted objective.Source: §3.1 · p. 4

3.2 Projection direction and gate

  • Conflict-only projection removes an opposing component while retaining aligned candidates; the always gate also removes aligned candidates’ parallel component.When projection applies, the combined gradient preserves the protected gradient’s first-order descent rate, but the unprotected loss need not decrease and optimizer transformations can change the update direction.Source: §3.2 · p. 5, §3.3 · p. 6

3.3 How protection interacts with the mixing weight

  • For conflicting gradients, the mixing weight scales different components depending on which gradient is protected, while the gate—not the weight—determines whether aligned components are retained.With one protection choice, the weight scales the candidate and its correction together; with the other, it scales the protected gradient while the correction is independent of the weight.Source: §3.3 · pp. 5–6

3.4 Accumulation and agent training

  • CORD accumulates the weighted objective gradients over each optimizer mini-batch, then projects once at the update scale because projection is nonlinear in the gradient pair.The agent recipes protect GRPO by projecting OPD, or protect OPD by projecting GRPO; their host objectives and weighting conventions are retained.Source: §3.4 · p. 6

4 Experiments

CORD adds a conflict-aware gradient-combination rule to three joint-training recipes and evaluates its benchmark gains and configuration choices. The results show improvements across all three recipes on ALFWorld and two on WebShop, while protection, gating, loss, and aggregation choices affect outcomes.

4 Experiments

  • The study tests whether CORD adds value across joint-training recipes, then examines protection, projection gating, loss construction, and agent behavior.It uses Skill-SD, GRPO+OPSD, and SDAR with Qwen2.5-3B-Instruct on ALFWorld and WebShop; final policies are evaluated in four passes after 150 updates.Source: §4 · p. 7, §4.1 · p. 7

4.2 RQ1: Incremental effectiveness

  • CORD raises ALFWorld pooled success for all three recipes: Skill-SD 76.4% to 77.9%, GRPO+OPSD 65.0% to 80.9%, and SDAR 73.8% to 77.7%.The respective gains are 1.5, 15.9, and 3.9 percentage points; GRPO+OPSD reaches the highest pooled success in the main comparison.Source: §4.2 · p. 7, Table 1 · p. 8
  • On WebShop, CORD improves Skill-SD success from 55.8% to 59.4% and SDAR success from 60.7% to 64.6%, while GRPO+OPSD changes from 64.8% to 64.5%.Across the shared configuration, CORD improves all three recipes on ALFWorld and two of three on WebShop.Source: §4.2 · pp. 7–8, Table 1 · p. 8

4.3 RQ2: The roles of projection and gradient choices

  • At λ=0.001, conflict-only projection beats no projection for either protected objective, while always projection yields 62.65% success protecting RL but 38.90% protecting distillation.At λ=0.005, all four CORD configurations fall between 61.25% and 62.10%, above the 56.75% unprojected result; the comparison is on WebShop.Source: Table 2 · p. 8, §4.3 · p. 9
  • At λ=0.005, ρd exceeds k3 on WebShop success, 58.95% versus 55.55%, whereas k3 exceeds ρd on ALFWorld success, 83.75% versus 79.11%.Changing the token objective changes the gradient CORD projects, and the effect of λ also varies by objective: raising it from 0.001 to 0.005 lowers k3 success from 59.35% to 55.55% but raises ρd success from 56.20% to 58.95%.Source: §4.3 · p. 9, Table 3 · p. 9
  • Mini-batch size and aggregation affect the projection comparison: RL-protected success varies by 0.78 percentage points across sizes 32–128, while distillation protection peaks at size 64.At size 64, protection-direction orderings differ across microbatch, global-token, and trajectory weighting, and all three paired intervals include zero.Source: §4.3 · p. 10, Table 4 · p. 10

4.4 RQ3: Case analysis of agent behavior

  • In the garment case, both projected policies complete all requested options and score 1.00, while the unprojected policy omits the color and scores 0.90.RL protection makes six steps without repeated selections; distillation protection repeats four selections and takes ten steps.Source: §4.4 · pp. 10–11, Figure 4 · p. 11

5 Conclusion

CORD combines two objective gradients while protecting a selected objective and making projection triggering explicit. The paper concludes that this reusable choice has incremental gains across the tested recipes, with configuration depending on protection, triggering, distillation loss, and aggregation.

5 Conclusion

  • CORD treats gradient combination as a reusable design choice, protecting either objective and explicitly selecting when projection is triggered.The paper reports incremental gains across three host recipes on ALFWorld and two of three on WebShop.Source: §5 · p. 11

Appendix Contents

A.1 On-policy and skill-conditioned distillation

  • On-policy distillation supplies token-level guidance on student-generated trajectories, while skill-conditioned methods provide contextual guidance from skills or privileged information.Skill-SD derives dynamic skills from completed trajectories and uses an importance-weighted reverse-KL objective with a periodically synchronized teacher.Source: §A.1 · p. 16

A.2 Evolving skills and reflection memory

  • Guidance can evolve through skill libraries, reflection memories, or trajectory-generated hindsight skills rather than remain fixed.CORD operates at the later combination of objective gradients and is described as orthogonal to whether guidance is fixed or evolving.Source: §A.2 · p. 16

A.3 Conflicting gradients and projection

  • CORD reuses a single-half-space projection: one objective gradient is protected while the other is projected, with conflict-only and always-project variants.The agent experiments examine protection, triggering, loss construction, and aggregation.Source: §A.3 · p. 17

A.4 Agent RL and GRPO-style policy optimization

  • Agent credit assignment and CORD address different stages: credit assignment shapes the reward-derived gradient, while CORD combines accumulated objective gradients.The gradients are combined after accumulation for an optimizer mini-batch.Source: §A.4 · p. 17

B.1 Datasets and Tasks

The appendix describes the ALFWorld and WebShop task settings and the shared implementations used to compare joint-training recipes. The recipes retain a fixed skill bank and differ in their distillation objectives, weights, and token weighting.

B.1 Datasets and Tasks

  • ALFWorld evaluates six household-task categories and pooled success on in-distribution games, requiring completion within 50 turns.The pooled rate weights categories by their number of evaluated tasks.Source: §B.1 · p. 18
  • WebShop requires finding and purchasing an instruction-matching product within 15 turns; evaluation reports graded score and strict success.Strict success requires a task score of 1; both measures use a 0–100 scale.Source: §B.1 · p. 18

B.2 Baselines

  • The three joint recipes combine a GRPO objective with token-level auxiliary losses, varying the distillation coefficient, confidence weighting, and OPD token objective.Base configurations disable projection.Source: §B.2 · p. 18, Table 6 · p. 19
  • SDAR confidence weights act before differentiation, whereas CORD uses the accumulated auxiliary gradient’s alignment with the GRPO actor gradient to gate projection.The SDAR weights emphasize sampled tokens where the skill-conditioned teacher has higher log-probability than the student.Source: §B.2 · p. 20
  • The shared skill bank stays fixed during training, and each Base–CORD pair shares its bank and skill-selection rule to isolate gradient combination.Household tasks select task-specific skills; WebShop uses instruction keywords alongside general strategies.Source: §B.2 · p. 20

B.3 Implementation and Evaluation Details

The implementation accumulates actor and auxiliary gradients across a complete mini-batch before applying CORD, then clips the combined gradient before AdamW. Final evaluation uses fixed tasks and the step-150 policy, with one training seed.

B.3 Implementation and Evaluation Details

  • The actor loss includes clipping safeguards, reference-policy KL, and optional entropy; CORD’s protected GRPO gradient differentiates this complete actor loss.The auxiliary loss is computed separately.Source: §B.3 · p. 21
  • CORD applies conflict handling to fully accumulated actor and distillation gradients; PCGrad instead projects each conflicting gradient against the other before summing.CORD’s combined gradient is clipped before the AdamW step.Source: §B.3 · p. 22
  • Final evaluation uses the policy after 150 updates without selecting the best intermediate checkpoint, and Base–CORD pairs share fixed task assignments and decoding settings.The final protocol uses four evaluation passes at temperature 0.4.Source: §B.3 · p. 23
  • Reported ± is population standard deviation across four evaluation passes, not variation across training seeds; final comparisons use one training seed.The projection identity alone does not establish descent after optimizer transformations or imply monotonic returns or convergence.Source: §B.3 · pp. 22–23
  • The WebShop comparison with published SDAR scores has different evaluation protocols: 500 fixed assignments here versus 128 validation tasks in SDAR.The paper notes that these differences limit direct comparison.Source: §B.3 · p. 23

B.4 Additional Experiments Across Model Scales

The supplementary 7B results show that CORD’s effect varies by host and task. The diagnostics and coefficient figures describe gradient and evaluation comparisons, but the supplied passages do not state their plotted outcomes.

B.4 Additional Experiments Across Model Scales

  • On Qwen2.5-7B-Instruct, ALFWorld success rises from 69.5% to 77.0% for Skill-SD and from 68.9% to 76.8% for SDAR; WebShop gains for SDAR but falls for Skill-SD.WebShop changes are +4.7 score points and +2.9 success points for SDAR, versus −2.9 score points and −1.8 success points for Skill-SD; these are single-seed results.Source: §B.4 · p. 23, Table 9 · p. 24

C.1 Motivation Diagnostics

  • The gradient diagnostic analyzes 2,741 WebShop/Qwen3-1.7B mini-batch records, of which 1,322 (48.2%) have negative RL–distillation cosine.Its panels compare conflict-only projection and coefficient reweighting, while a separate training comparison uses Qwen2.5-3B-Instruct.Source: §C.1 · p. 24

C.2 Token Objectives and Mixing Coefficients

  • Figure 5 reports WebShop task score alongside the success-rate curves, using RL protection and conflict-only projection; the supplied text gives no plotted values.The caption states that its checkpoints and evaluation protocol match Figure 3 and that error bars show standard deviation across decoding passes.Source: Figure 5 · p. 25, §C.2 · p. 25

C.3 Task-level Projection Intervention

The task-level intervention compares projection directions by locally reweighting fixed gradient pairs. Its reported relative update change is a diagnostic, not a result from retraining at each coefficient.

C.3 Task-level Projection Intervention

  • The intervention measures the relative difference between CORD and direct-sum descent directions using fixed gradients, checkpoints, and trajectories rather than retraining at each coefficient.It uses eight available ALFWorld source tasks and all 16 WebShop source tasks.Source: §C.3 · p. 25
  • Figure 6 compares protection directions under always projection, connecting the two directions for each source task and reusing gradients across coefficients.Its horizontal axis reports relative change in the combined update; U0 and Ut denote dsum and dCORD.Source: §C.3 · p. 25, Figure 6 · p. 26
  • With fixed gradients, the distillation-protection correction is independent of the positive mixing coefficient, but the retained distillation term still scales with it.The stated correction relationship also holds on the conflicting branch of conflict-only projection.Source: §C.3 · p. 26

C.4 Sensitivity to Optimization Choices

The ALFWorld continuation study varies optimizer mini-batch size and loss aggregation under a fixed protocol. Protection rankings depend on both choices, and the reported paired intervals do not establish a consistent aggregation winner.

C.4 Sensitivity to Optimization Choices

  • The continuation starts from a shared ALFWorld/Qwen2.5-3B checkpoint and AdamW state, then runs five rounds with one training seed.Evaluation uses 512 attempts across 91 in-distribution tasks; 95% task-bootstrap intervals describe evaluation uncertainty for this seed.Source: §C.4 · pp. 26–27
  • At B=128, distillation protection is 5.95 percentage points below no projection, with a paired 95% interval of [−11.13, −0.96].Under L2, RL-protected success spans 0.78 percentage points across B∈{32, 64, 128}, while distillation protection peaks at B=64.Source: §C.4 · p. 27
  • Distillation protection leads by 3.71 points under global-token averaging, while RL protection leads by 4.35 under trajectory averaging at B=64.All three paired intervals include zero, so the ordering remains descriptive; aggregation reweights gradients before projection.Source: §C.4 · p. 28

C.5 Training Dynamics

Single-seed training trajectories show higher late-training success and shorter episodes with CORD in the reported WebShop and ALFWorld settings. On ALFWorld, CORD also raises token entropy while slightly reducing response length.

C.5 Training Dynamics

  • The training-trajectory comparisons use matched code revision, active configuration, and seed within each Base/CORD pair, but each method has one seed.The plots show single-seed on-policy trajectories and are separate from final evaluations.Source: §C.5 · p. 28, Figure 8 · p. 29, Figure 9 · p. 30
  • In WebShop, success rises from 50.0% to 57.1% as episode length falls from 8.58 to 7.48 turns; in 7B ALFWorld, success rises from 61.0% to 76.0% as length falls from 30.67 to 25.34 turns.Mean reference-KL estimates remain close within each pair: 0.188 versus 0.189 on WebShop and 0.274 versus 0.271 on ALFWorld.Source: §C.5 · p. 28, Figure 8 · p. 29
  • On 3B ALFWorld over iterations 101–148, token entropy increases from 0.780 to 0.925 nats while mean response length decreases from 137.1 to 133.3 tokens.Entropy is measured on each policy’s own visited states and does not by itself measure exploration quality or establish a final-evaluation gain.Source: §C.5 · p. 28, Figure 9 · p. 30

C.6 Computational cost

CORD adds gradient-combination work to each optimizer mini-batch, with measured per-mini-batch overhead and a smaller estimated iteration-level multiplier for the recorded WebShop workloads. The section also reports repeated-evaluation significance tests for Table 1 comparisons.

C.6 Computational cost

  • CORD accumulates both objective gradients over a complete PPO mini-batch, requiring O(d) arithmetic and storage per mini-batch.Packed reductions communicate scalar inner products and norms with message size that does not grow with parameter count d.Source: §C.6 · p. 31
  • Sharing student forward computation avoids duplicate outputs but does not make two objective-gradient evaluations equivalent to one backward pass on their summed loss.Implementation opportunities do not guarantee a fixed wall-clock improvement; the component-based estimates are not measurements of a complete optimized training run.Source: §C.6 · pp. 31–32
  • The summed per-mini-batch median time is 1.78× that of an ordinary Base that backpropagates the summed loss once.The measurement includes student forward and backward computations, communication, gradient combination, clipping, and AdamW.Source: §C.6 · p. 32
  • The estimated iteration multiplier for two recorded WebShop workloads is 1.20–1.21×, with an estimated Base actor share of 25–27%.The estimates hold shared-stage costs fixed and assume ALFWorld replay actor ratios transfer to WebShop workloads.Source: §C.6 · p. 32, Table 10 · p. 32

C.7 Statistical Significance

  • Table 11 reports two-sided Welch tests on four evaluation-pass means per checkpoint, with Holm correction across 27 ALFWorld and WebShop comparisons.These tests summarize decoding-pass variation at one training seed.Source: §C.7 · p. 32, Table 11 · p. 33

C.8 Additional Case Analyses

Additional case analyses show that policy behavior differs in repetition, attribute completion, and task-subgoal completion, with trajectory-length rankings varying by case. Token-level analysis finds context-dependent probability changes after projection, including an adverse change for an unlisted token.

C.8 Additional Case Analyses

  • In WebShop, RL protection makes eight repeated option clicks in a 14-step rug purchase, while distillation protection uses six steps; for a garment, RL protection scores 0.80 by omitting the requested size while the other policies score 1.00.All policies buy the same rug product, and all finish the garment case in five steps.Source: §C.8 · p. 33, Figure 10 · p. 34
  • In ALFWorld, both projected policies complete the heating-and-placement sequence, taking eight steps under RL protection and sixteen under distillation protection; for the second delivery, distillation protection takes 11 steps versus 25 under RL protection.The unprojected policy fails to heat an egg within 50 steps in the first case and ends with only one delivered clock in the second.Source: §C.8 · p. 35, Figure 11 · p. 36

C.9 Token-Level Effects of Projection

  • After one update on the same conflicting batch, CORD with OPD protection retains more probability on “he” in “heather” and “men” than DirectSum, but token changes vary by context.At the repeated size choice, CORD reverses DirectSum’s increase, and the analysis also reports an adverse increase for an unlisted token.Source: §C.9 · p. 37, Figure 12 · p. 38

Related papers

Loading 2610.29965v1…