PaperlayerView exact revision on arXiv

Source-linked AI summary

Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models

Zongji Yu, Wenshui Luo, Yiliu Sun, Hao Fang, Runmin Cong, Chaochao Lu, Chen Gong

arXiv:2605.25443v1cs.CL

TL;DR

Multi-domain GRPO-style post-training can suffer inter-domain interference while underusing transferable knowledge among rollouts. MCPO adds rollout-level contrastive learning for cross-domain sharing and in-domain consolidation, and experiments report consistent improvements over GRPO and DAPO across five domains. The study frames its evidence as a representative five-domain evaluation rather than an exhaustive comparison of all domain mixtures or post-training variants.

  • Problem

    GRPO-style multi-domain optimization can cause inter-domain interference, while existing approaches often neglect knowledge sharing among compatible domains.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    MCPO builds a structured rollout-level contrastive objective that attracts compatible correct rollouts, consolidates in-domain correct rollouts, separates incorrect rollouts, and complements GRPO.

    Source: §4 · p. 6

  • Results

    MCPO consistently improves corresponding GRPO and DAPO baselines across Math, Code, Science, ToolUse, and Safety, with gains over baselines reported on Qwen3-4B and Qwen3-8B.

    Source: §5.2 · p. 9, §6 · p. 11

  • Takeaways & Limitations

    Structured knowledge sharing and in-domain consolidation extend multi-domain RL beyond interference alleviation toward a representation space accommodating diverse domain knowledge.

    Source: §1 · p. 2, §6 · p. 11

    The evaluation covers a representative five-domain setting and selected backbones rather than every possible domain mixture, recent post-training variant, or implementation baseline.

    Source: §A.4 · p. 24

Abstract

from arXiv · show

Post-training has significantly enhanced the reasoning capability of Large Reasoning Models (LRMs), especially with Reinforcement Learning (RL) like Group Relative Policy Optimization (GRPO). However, GRPO-style RL methods in multi-domain settings often fail to achieve consistent improvements across all domains due to inherent interference in policy optimization. Prior studies on multi-domain RL primarily focus on alleviating cross-domain interference, while often neglecting the pivotal role of knowledge sharing, which we argue is the key to transforming cross-domain interactions from harmful competition into beneficial transfer. To address this limitation, we propose Multi-domain Contrastive Policy Optimization (MCPO), which analyzes the structural relationships among rollouts and promotes cross-domain knowledge sharing and in-domain knowledge consolidation in a contrastive manner. Specifically, for a given prompt, MCPO identifies transferable reasoning trajectories from other domains as positive examples, while treating incorrect rollouts as negative ones. It then encourages consistent representations for positive pairs and pushes negative pairs apart, thereby facilitating knowledge transfer and reducing interference. Moreover, MCPO aligns intra-domain correct rollouts to build a consolidated representation space. In this way, MCPO contrastively learns a harmonious representation space that can accommodate diverse multi-domain knowledge. Empirical results show that MCPO improves the reasoning capabilities of LRMs across multiple domains and even outperforms single-domain training in some cases. Code is available at https://github.com/Maricalce/MCPO.

1. Introduction

Multi-domain post-training must preserve reasoning abilities across heterogeneous domains, but GRPO-style optimization can cause interference and neglect beneficial knowledge sharing. MCPO addresses this by contrastively organizing rollout relationships for cross-domain transfer and in-domain consolidation.

  • Multi-domain training can improve one domain while degrading another because shared optimization causes inter-domain interference.Source: §1 · pp. 1–2
  • Correct rollouts can preserve transferable reasoning structures, while incorrect rollouts provide negative examples for contrastive learning.Source: §1 · p. 2
  • MCPO constructs a structured contrastive pool from mixed-domain rollouts, using compatible cross-domain correct trajectories for sharing and in-domain correct trajectories for consolidation.Source: §1 · p. 2
  • MCPO combines its contrastive loss with within-prompt GRPO to add supervision for positive and negative relationships across prompts and domains.Source: §1 · p. 2
  • MCPO improves reasoning across multiple domains and can outperform single-domain training in some cases.Source: §1 · p. 2

2. Related Work

Prior multi-domain RL methods mainly reduce harmful interference through sampling, normalization, weighting, curriculum, or gradient balancing. This work instead emphasizes positive structural associations among rollouts to support transfer and consolidation across domains.

  • Multi-domain RL faces interference and task imbalance because heterogeneous prompt groups update one policy under different rewards, difficulties, and rollout dynamics.Source: §2 · p. 3
  • Existing methods mitigate conflicts through mixture design, reward normalization, task reweighting, curriculum or sequential training, and gradient balancing.Source: §2 · p. 3
  • Contrastive learning increases similarity between positive pairs and decreases similarity between negative pairs, matching the use of diverse correct rollouts as mutually augmented examples.Source: §2 · p. 3
  • Existing contrastive integrations mostly operate within prompts, whereas MCPO extends structural rollout relationships across domains for knowledge sharing and in-domain consolidation.Source: §2 · p. 3

3. Preliminaries

The preliminaries define multi-domain RL as updating one policy with domain-specific rewards while seeking both higher average reward and compatible domain abilities. They then describe GRPO’s within-prompt clipped optimization and its limitation for cross-prompt knowledge sharing.

3.1. Problem Definition

  • Multi-domain RL updates one policy with domain-specific rewards and seeks compatible coexistence of reasoning abilities across heterogeneous domains, not only higher average reward.Source: §3.1 · p. 3
  • Each domain supplies prompts and targets, while evaluators assign binary rewards indicating whether sampled rollouts are correct.Source: §3.1 · p. 3
  • Mixed-domain training jointly uses examples from different domains in one step and updates the shared policy toward average reward over the mixture.Source: §3.1 · pp. 3–4

3.2. Group Relative Policy Optimization

  • GRPO samples multiple rollouts per prompt, computes relative advantages within each group, and broadcasts sequence-level advantages to rollout tokens.Source: §3.2 · p. 4
  • GRPO’s clipped token-level objective provides efficient within-prompt credit assignment and stable optimization without a heavy critic network.Source: §3.2 · p. 4
  • Because GRPO computes advantages only within prompts and aggregates losses across domains, it neither consolidates successful in-domain structures nor enables cross-domain sharing.Source: §3.2 · p. 4

4. MCPO: Multi-domain Contrastive Policy Optimization

MCPO augments GRPO with a rollout-level contrastive objective that treats compatible correct trajectories as positives and incorrect trajectories as negatives. It combines weighted cross-domain sharing with in-domain consolidation in a normalized representation space.

  • MCPO maps verifier-labeled rollouts into a normalized representation space and jointly optimizes GRPO with contrastive supervision for sharing, consolidation, and negative separation.Source: §4 · pp. 5, 7

Cross-domain knowledge sharing

  • Cross-domain sharing weights correct rollout pairs by prompt-prototype compatibility, giving more similar prototypes stronger attraction.Source: §4 · pp. 6–7

4. MCPO: Multi-domain Contrastive Policy Optimization

  • Correct rollouts from compatible domains and the same domain form positive examples, while incorrect rollouts form negative examples.Source: §4 · p. 6

In-domain knowledge consolidation

  • In-domain consolidation aligns correct rollouts from the same prompt and from different prompts within the same domain to preserve recurring reasoning patterns.Source: §4 · pp. 6–7
  • The cross-domain and in-domain objectives normalize their positive-example mass and balance their contributions before combining with GRPO.Source: §4 · pp. 6–7

5. Experiments

Experiments evaluate MCPO across five reasoning domains and multiple backbones, showing consistent gains over GRPO-style baselines and supporting its contrastive transfer mechanism.

5.1. Experimental Setup

  • MCPO is evaluated on Qwen3-4B, Qwen3-8B, DeepSeek-R1-Distill-Qwen-7B, and Llama3.1-8B-Instruct across a five-domain RL post-training setup.The setup includes mathematical reasoning, code generation, scientific question answering, tool use, and safety alignment.Source: §5.1 · pp. 7–8

5.2. Main Results

  • MCPO-DAPO raises Qwen3-8B average accuracy from 51.68 to 55.87, while MCPO-GRPO raises it from 50.51 to 54.22.On Qwen3-4B, the corresponding gains are 49.24 to 52.60 for MCPO-DAPO and 47.95 to 51.69 for MCPO-GRPO.Source: §5.2 · p. 9
  • MCPO-DAPO improves Math and Science by 4.60 and 2.10 points, Code by 5.74 points, and ToolUse and Safety by 4.72 and 3.50 points.The evaluation covers Math, Code, Science, ToolUse, and Safety using domain-specific benchmarks and overall average accuracy.Source: §5.1 · p. 8, §5.2 · p. 9
  • Compared with MCPO-DAPO, CLIPO leaves a 5.41-point gap on Qwen3-8B, while MGS and MT-GRPO reach averages of 51.77 and 52.09.These comparisons distinguish cross-domain knowledge sharing from within-prompt contrastive feedback, gradient surgery, and task reweighting.Source: §5.2 · p. 9

5.3. Algorithmic Analyses

  • Correct positive examples increase from roughly 1.7 to above 2.3, their ratio from about 0.47 to 0.60, and positive weight from about 0.75 to 0.88 during training.Cross-domain positive logits rise from about 6.0 to 8.8, negative logits fall from around 2.0 to 1.1, and the margin widens from about 4.0 to 7.6.Source: §5.3 · pp. 9–10, Figure 2 · p. 10
  • MCPO brings Math and Science representations closer while separating Safety and ToolUse, and MCPO-DAPO maintains higher average reward and policy entropy than DAPO.The contrastive objective therefore combines transferable-domain attraction with separation of unrelated domains and incorrect rollouts.Source: §5.3 · p. 10, Figure 3 · p. 10
  • Ablations show DAPO falls from 55.64 in single-domain training to 51.68 in mixed-domain training, whereas MCPO reaches 55.87.Additional panels support contrastive-head warmup, joint policy-head optimization, a 512-response pool, and transfer under GRPO from 40.18 to 44.32.Source: §5.3 · p. 11, Table 2 · p. 11

6. Conclusion

MCPO extends multi-domain RL by organizing rollout relationships for knowledge sharing and consolidation, improving performance across diverse domains.

  • MCPO models verifier-labeled rollout relationships so compatible correct rollouts share knowledge, same-domain correct rollouts consolidate knowledge, and incorrect rollouts become negatives.Experiments span Math, Code, Science, ToolUse, and Safety and improve corresponding GRPO and DAPO baselines.Source: §6 · p. 11
  • MCPO learns a representation space intended to accommodate diverse domain knowledge rather than only alleviate interference.The conclusion frames this as an extension from conflict reduction toward structured cross-domain interaction.Source: §6 · p. 11

A.1.1. Baseline Implementation

The baseline implementations preserve comparable RL training settings while varying reward handling, contrastive scope, gradient updates, or task scheduling.

  • GRPO uses a vanilla PPO-style clipped surrogate objective with group-relative advantages and token-level aggregation, without KL reward shaping.It serves as the direct reference for gains from cross-rollout relational regularization.Source: §A.1.1 · p. 16
  • DAPO retains the GRPO pipeline but switches to DAPO reward management and long-response handling while keeping other training factors unchanged.This isolates reward shaping from MCPO-style contrastive regularization.Source: §A.1.1 · p. 16
  • CLIPO adds an auxiliary contrastive objective focused on within-prompt or in-domain alignment rather than full cross-domain transfer.It tests whether gains arise from adding a contrastive head alone or from MCPO's relational design.Source: §A.1.1 · p. 16
  • MGS applies task-wise gradient surgery across domains without changing the reward or sampling pipeline.This compares MCPO with an optimization-side multi-task baseline targeting destructive gradients.Source: §A.1.1 · p. 16
  • MT-GRPO uses task-adaptive curriculum and sampling with rejection sampling and FAMO-related components to emphasize weaker tasks.The comparison tests adaptive scheduling against relational supervision.Source: §A.1.1 · p. 16

A.1.2. MCPO Implementation

MCPO adds a lightweight contrastive head and structured regularization to a GRPO-based setup, with jointly trained five-domain rollouts and a warmup schedule.

  • MCPO is implemented on VERL with GRPO optimization settings plus an additional contrastive head and structured cross-domain regularization.The implementation uses algorithm_type=grpo and adv_estimator=grpo.Source: §A.1.2 · pp. 16–17
  • Training jointly covers Math, Code, Science, ToolUse, and Safety with Qwen3-8B, 128 prompts per batch, and eight sampled rollouts per prompt.Training runs for three epochs on eight H200 GPUs with 2048-token prompts and 8192-token responses.Source: §A.1.2 · p. 17
  • The policy uses a vanilla PPO-style clipped objective with KL reward shaping and actor-side KL loss disabled, plus an entropy coefficient of 0.001.Padding removal and gradient checkpointing are enabled for the actor model.Source: §A.1.2 · p. 17
  • MCPO uses a 256-dimensional projection head, contrastive coefficient 0.1, in-domain balancing coefficient 0.5, and temperature 0.1.Responses with reward at least 0.5 are positive anchors, and the cross-positive attraction coefficient is 0.5.Source: §A.1.2 · p. 17
  • A 100-mini-batch contrastive warmup trains the projector while protecting the backbone from early noisy contrastive gradients.The contrastive pool is built from current actor representations rather than reused rollout-cached representations.Source: §A.1.2 · p. 17

A.1.3. Prompt Construction

The prompt construction uses domain-specific system and user-message formats so each task preserves its required reasoning or answer protocol.

  • Math prompts request step-by-step reasoning followed by a boxed final answer.Source: §A.1.3 · p. 17
  • Code prompts require a Python program only, while preserving the original programming task statement.Source: §A.1.3 · pp. 17–18
  • Science prompts present four options and require a letter-only answer inside the specified reasoning-and-answer format.Source: §A.1.3 · p. 18
  • ToolUse prompts place task instructions, available tools, documentation, and constraints directly in the user message without an additional system instruction.Source: §A.1.3 · p. 18
  • Safety prompts ask the model to select the safer response between two candidates using an exact output label.Source: §A.1.3 · p. 18

A.1.4. Domain-specific Reward Computation

The paper evaluates domain-specific verifiable rewards and then analyzes MCPO through ablations, representation geometry, and Math-to-Science case studies. These analyses attribute the gains to structured rollout-level knowledge sharing and consolidation, while noting that evaluation covers only a representative five-domain setting.

A.1.4. Domain-specific Reward Computation

  • The domain-specific reward system assigns binary correctness rewards using mathematical equivalence, code execution, answer matching, tool-call matching, or safety-label verification.Source: §A.1.4 · pp. 18–19

A.2.1. Ablation Diagnostics Support the Structured Contrastive Design

  • MCPO's ablation diagnostics isolate cross-domain knowledge sharing, in-domain consolidation, a trainable representation head, and rollout-pool diversity as components of its structured contrastive design.The default contrastive pool contains 512 responses; on DeepSeek-R1-Distill-Qwen-7B, MCPO improves GRPO from 40.18 to 44.32, versus 33.00 for the base model.Source: §A.2.1 · p. 19

A.2.2. Representation Geometry Reflects Knowledge Sharing Beyond Interference Alleviation

  • MCPO's representation geometry combines compact in-domain clusters, preserved domain identity, and proximity between domains with transferable reasoning structures rather than global mixing or rigid isolation.Figure 4 compares GRPO, DAPO, CLIPO, MGS, MT-GRPO, and MCPO on the same held-out mixed-domain test set with identical t-SNE settings.Source: §A.2.2 · pp. 19–20, Figure 4 · p. 20

A.2.3. Case Study: Math-to-Science Knowledge Sharing on SciBench Questions

  • In SciBench case studies, MCPO-DAPO adapts Math structures to Science semantics, producing positive work for the Hooke-law problem and ⟨V⟩ = E/3 = 3.33 eV for the quartic-potential problem.The cases are presented as qualitative evidence that transferable Math reasoning structures can support quantitative Science problem solving.Source: §A.2.3 · pp. 21, 22, 24

A.4. Limitations

  • The study focuses on five domains and selected Qwen-family and heterogeneous backbones, so it does not exhaust all domain mixtures, post-training variants, or implementation baselines.The authors frame the comparison as testing structured knowledge sharing against prompt-local optimization and conflict-mitigation mechanisms.Source: §A.4 · p. 24

Related papers

Loading 2605.25443v1…