PaperlayerView exact revision on arXiv

Source-linked AI summary

DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignment

Henan Sun, Zehua Li, Haitao Hu, Qifan Zhang, Jianfeng Zhang, Nuo Chen, Jia Li

arXiv:2609.27572v1cs.LGcs.AI

TL;DR

Rule-based and learned reward systems can struggle to evaluate evolving LLM reasoning reliably. DCRL evolves reward rubrics and updates reward models alongside policy models. Across reasoning domains, it outperforms the reported baselines, with Qwen3-4B surpassing Qwen3-32B and approaching Qwen3-235B on evaluated benchmarks.

  • Problem

    Rule-based rewards may miss quality in complex or open-ended tasks, while static reward models can become mismatched with evolving policies and contribute to unstable optimization and reward hacking.

    Source: §1 · pp. 1–2

  • Method

    DCRL dynamically refines reward rubrics with syllogistic-logic prompts and synchronizes reward-model parameters with the updated policy model.

    Source: §1 · p. 3, §3.2 · pp. 5–6

  • Results

    DCRL consistently outperforms rule-based and static reward-model baselines across diverse reasoning domains, with Qwen3-4B surpassing Qwen3-32B and approaching Qwen3-235B on evaluated benchmarks.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that policy–reward co-evolution is effective for improving performance and generalization across the evaluated reasoning tasks.

    Source: §5 · p. 9

    The theoretical analysis assumes prompt smoothness: small prompt changes induce small perturbations in the reward reasoning distribution.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Reinforcement learning (RL) has emerged as a key paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing reward systems, such as rule-based and reward-model-based, often exhibit issues such as unstable optimization and reward hacking. In this work, we revisit the general reasoning of LLMs from a geometric perspective, conceptualizing it as a coupled manifold composed of three interdependent sub-manifolds: logical deduction, evaluation, and representation. Based on this perspective, response generation in RL can be interpreted as a decoupling process from the evaluation manifold, while reward estimation corresponds to a decoupling process from the logical deduction manifold. The limitations of rule-based and reward-model RL systems can be geometrically interpreted as the mismatch of policy-reward manifolds during RL process. To address the aforementioned misalignment, we propose Decoupling and Coupling Reinforcement Learning (DCRL) framework, which incorporates two key components: (1) a syllogistic logic-based prompt evolution mechanism that dynamically refines reward rubrics to enhance the expressiveness of the reward manifold; and (2) a policy-reward re-coupling mechanism that jointly updates the reward and policy models, ensuring consistent evaluation and mitigating manifold mismatch during training. Theoretical analysis and extensive experiments across multiple reasoning domains demonstrate that DCRL consistently outperforms both rule-based and reward-model baselines. Notably, a Qwen3-4B model trained under DCRL surpasses a Qwen3-32B baseline and approaches the performance of a Qwen3-235B model, highlighting superior effectiveness and generalization in RL.

1 Introduction

The introduction frames reward reliability as a challenge in RL for LLM reasoning and models reasoning as coupled logical-deduction, evaluation, and representation sub-manifolds. It presents DCRL as a way to evolve reward rubrics and recouple reward and policy models.

1 Introduction

  • Rule-based rewards can miss response quality in complex or open-ended tasks and invite reward hacking, while learned reward models can be over-optimized and misaligned with task objectives [Chen et al., 2024; Miao et al., 2024; Rafailov et al., 2023; Wen et al., 2024].The cited discussion notes that rule-based systems may reward superficial rule satisfaction and reward models may provide inaccurate signals.Source: §1 · pp. 1–2
  • The paper models reasoning as coupled logical-deduction, evaluation, and representation sub-manifolds, with response generation and reward estimation decoupling from different sub-manifolds.Response generation decouples from evaluation, whereas reward estimation decouples from logical deduction.Source: §1 · p. 2
  • A static reward rubric may not capture changing evaluation criteria as policy reasoning evolves, and a static reward model may become misaligned with the policy's output distribution.The proposed motivations are to evolve the rubric and update the reward model alongside the policy.Source: §1 · pp. 2–3
  • DCRL evolves reward rubrics through syllogistic-logic prompts and re-integrates decoupled evaluation and representation sub-manifolds through RL-based coupling.The authors describe the resulting optimization as more stable and its performance improvements as consistent.Source: §1 · p. 3

Our Contributions

  • Across mathematics, code, and commonsense, DCRL outperforms rule-based and static reward-model baselines; Qwen3-4B surpasses Qwen3-32B and approaches Qwen3-235B.The authors present this as evidence of DCRL's effectiveness and generalization.Source: §1 · p. 3

2 Related Work

The related work groups RL reward systems into rule-based systems and learned reward models, describing their use in reasoning and code tasks.

Rule-based systems

Reward model systems

  • Learned reward systems include outcome, preference, and process reward models, with examples spanning human-AI preference data, trajectory-aware supervision, execution feedback, specialized verifiers, and generative critique [Liu et al., 2025a; Zou et al., 2025; Li et al., 2025; Liu et al., 2025b; Liu et al., 2025c].The paper describes their efficiency and ease of integration as making them a practical foundation for reward-guided policy optimization.Source: §2 · p. 4

3 Methodology

The methodology motivates DCRL through policy–reward manifold mismatch, implements dynamic prompt evolution and parameter synchronization, and analyzes bounds on prompt updates and the reasoning-manifold gap.

3.1 Motivation

  • The motivation is that rule-based rewards can overlook the model's evaluation sub-manifold, while static reward models can mismatch the evolving policy and destabilize optimization.The proposed response is to evolve reward prompts and update the reward model with the improved policy.Source: §3.1 · p. 4

3.2 DCRL Framework

  • DCRL uses fixed axiomatic prompts, domain-adaptable theorem prompts, and dynamically updated corollary prompts; at each epoch, it synchronizes reward-model parameters with the updated policy model.Prompt evolution refines evaluation criteria, while parameter synchronization is intended to reduce structural mismatch.Source: §3.2 · pp. 5–6

3.3 Theoretical Analysis

  • Under prompt smoothness, Theorem 3.2 bounds the prompt-evolution step by a constant δ, using average advantage, training accuracy, and the reward model's rewriting operator.The prompt-smoothness assumption states that small prompt changes induce small perturbations in the reward reasoning distribution.Source: §3.3 · p. 6
  • Theorem 3.3 states that the reasoning-manifold gap between policy and reward models remains uniformly bounded throughout training.The paper identifies the gap with symmetric KL divergence between their reasoning distributions and says the bound supports stable reward evaluation.Source: §3.3 · pp. 6–7

4 Experiments

Experiments evaluate DCRL across diverse reasoning benchmarks, comparing performance, policy–reward manifold alignment, and ablated variants. DCRL-4B surpasses Qwen3-4B and Qwen3-32B and approaches Qwen3-235B, while showing stronger alignment and sustained gains than the ablations.

4.1 Experiment Setup

  • The evaluation covers 10 benchmarks across math, code, and commonsense, with in-distribution and out-of-distribution testing.Baselines include zero-shot Qwen3 models, rule-based and static reward systems, and three other representative systems; pass@5 is reported, with Linear CKA and Procrustes Distance for alignment.Source: §4.1 · p. 7

4.2 Performance Comparison

  • DCRL-4B surpasses Qwen3-4B and Qwen3-32B across evaluated benchmarks and remains competitive with Qwen3-235B.The comparison spans math, code, and commonsense benchmarks.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 Manifolds Alignment Analysis

  • Across OlyBench, CodeContest, and BBEH, DCRL achieves generally higher Linear CKA and lower Procrustes Distance than the static-reward setting.These metrics indicate stronger structural similarity and reduced geometric discrepancy between policy and reward reasoning manifolds.Source: §4.3 · p. 8, Figure 2 · p. 9

4.4 Ablation Study

  • Full DCRL consistently outperforms the static-reward and rule-based ablations across OlyBench, CodeContest, and BBEH.DCRL maintains an upward performance trend, while both ablated variants’ gains gradually plateau.Source: §4.4 · p. 9

5 Conclusion

The paper frames LLM reasoning as coupled logical-deduction, evaluation, and representation manifolds, interpreting reward-system limitations as policy–reward mismatch. DCRL combines dynamic reward-rubric evolution with policy–reward re-coupling; on evaluated benchmarks, DCRL-trained Qwen3-4B surpasses Qwen3-32B and approaches Qwen3-235B.

5 Conclusion

  • The paper models LLM reasoning as coupled logical deduction, evaluation, and representation manifolds, with reward-system limitations arising from policy–reward mismatch.It presents this geometric framing as a perspective on reasoning and reward modeling.Source: §5 · p. 9
  • DCRL integrates dynamic reward-rubric evolution with policy–reward re-coupling to support more stable and aligned optimization.The framework is presented as policy–reward co-evolution.Source: §5 · p. 9
  • On evaluated benchmarks, DCRL-trained Qwen3-4B surpasses Qwen3-32B and approaches Qwen3-235B.The paper reports improved performance and generalization over conventional rule-based and reward-model approaches.Source: §5 · p. 9

A.1 Theoretical Proof about Theorems 3.2

The proof bounds prompt-representation changes caused by prompt evolution. It uses a limit on modified tokens and a bound on token-embedding norms to establish the perturbation bound for Theorem 3.2.

A.1 Theoretical Proof about Theorems 3.2

  • The proof represents the full prompt as concatenated token embeddings and measures change by the sum of changes to modified-token embeddings.The prompt-evolution operator is treated as acting on this embedding representation.Source: §A.1 · p. 14
  • The resulting bounded prompt perturbation establishes Theorem 3.2.The proof identifies the bound as the upper bound on the prompt-evolution step.Source: §A.1 · p. 15
  • At most m modified tokens, each with bounded embedding norm M, limit the total prompt perturbation to a constant.Each modified token contributes at most 2M to the total embedding change.Source: §A.1 · pp. 14–15

A.2 Theoretical Proof about Theorems 3.3

The proof tracks policy–reward reasoning-manifold discrepancy across training. Epoch-end synchronization makes the distributions identical before prompt evolution, and bounded prompt changes keep their discrepancy bounded afterward.

A.2 Theoretical Proof about Theorems 3.3

  • Epoch-end parameter synchronization makes the policy and reward reasoning distributions identical before prompt rewriting, so their discrepancy is then zero.The proof uses the synchronization strategy from the training algorithm.Source: §A.2 · p. 15
  • Prompt evolution is the remaining source of discrepancy after synchronization, and its bounded change yields a bounded divergence between the reasoning distributions.The proof bounds divergence using the assumption that nearby prompts induce nearby reward reasoning distributions.Source: §A.2 · pp. 15–16
  • The proof concludes that the policy and reward reasoning manifolds remain within a bounded discrepancy throughout DCRL training.Source: §A.2 · p. 16

B.1 Dataset Pre-processing

The preprocessing pipeline standardizes heterogeneous benchmarks for a shared RL framework while normalizing answers to make reward evaluation reflect task correctness. Dataset-specific formatting and verification procedures accommodate mathematical, coding, and other reasoning tasks.

B.1 Dataset Pre-processing

  • A unified chat-style schema and shared dataloader constraints support processing heterogeneous benchmarks, while dataset-specific normalization aims to make rewards reflect semantic correctness rather than formatting.The pipeline also uses prompt-level formatting, backend fallbacks, and prewarm validation to support answer extraction and reproducibility.Source: §B.1 · p. 16
  • GSM8K uses numeric-answer supervision, with commas and currency symbols removed from predictions and labels before evaluation.The benchmark contains approximately 8.5K grade-school mathematical word problems.Source: §B.1 · p. 16
  • AIME24 and AIME25 use normalized exact symbolic matching, while OlyBench prompts require answers in a specified final-answer format.Serialized list-like AIME outputs are converted to canonical scalar forms; OlyBench prompts are regenerated when source data or formatting instructions change.Source: §B.1 · p. 17
  • CodeContest uses a prewarm check for external test-case access and evaluates solutions with public, private, and generated tests under bounded time limits.LiveCodeBench solutions are tested in a sandbox using question-linked unit tests after deterministic train–validation partitioning.Source: §B.1 · p. 17
  • BBEH answers are normalized to canonical forms for exact-match evaluation, while MMLU-Pro outputs are normalized to option labels for exact-agreement rewards.GPQA preprocessing removes incomplete entries, randomizes distractors, balances answer labels, and formats prompts with final-answer instructions.Source: §B.1 · p. 17

B.2 Experiments Pre-processing in Section 4.3

The section describes an alignment analysis comparing co-evolutionary and non-co-evolutionary reward training using output embeddings, CKA, and Procrustes distance. It also gives example DCRL prompts that structure reward criteria hierarchically and specify answer formats.

B.2 Experiments Pre-processing in Section 4.3

  • The alignment analysis compares co-evolutionary and non-co-evolutionary training with step-wise Linear CKA and Procrustes distance.Higher CKA indicates greater structural similarity; lower Procrustes distance indicates better geometric consistency.Source: §B.2 · pp. 17–19
  • Rollout records pair policy and reward responses with training indices; alignment metrics are aggregated across valid steps and plotted against normalized training progress.The section describes mean CKA and mean Procrustes distance for summarizing the training trajectory.Source: §B.2 · pp. 18–19
  • Policy and reward outputs are embedded using TF-IDF followed by truncated SVD, with hashed bag-of-tokens as a fallback, then compared as paired output-space approximations of the reasoning manifolds.These metrics assess alignment between empirical textual projections, not the full latent manifolds directly.Source: §B.2 · p. 18

C Example Prompts for DCRL

  • Example task prompts constrain final-answer formats, and one reward prompt treats execution-based evaluation as primary while making the score auxiliary.Source: C Example Prompts for DCRL · pp. 19–20
  • The example syllogistic prompts separate immutable axioms from domain-dependent theorems and tactical corollaries, which are updated using scoring experience.The prompt asks for updated theorem and corollary rules in a strict JSON response.Source: C Example Prompts for DCRL · pp. 19–20

D Limitations

DCRL’s limitations concern uncertain transfer to agentic RL and added computational costs, prompt-design sensitivity, and update stability; the paper also reports how generative AI was used and checked.

D Limitations

  • DCRL’s applicability to agentic RL remains unclear because adapting it to dynamic, multi-step decisions is non-trivial.The current formulation is designed primarily for RL in LLMs.Source: D Limitations · p. 20
  • Syllogistic prompt evolution adds computational costs, and prompt design and update stability may affect training robustness and efficiency.Source: D Limitations · p. 20

AI Use Statement

  • Generative AI assisted with translation, code implementation, and experiment feedback; two authors verified and tested LLM-generated code.The authors state that they reviewed all AI-assisted work and take responsibility for the final content.Source: AI Use Statement · p. 20
Loading 2609.27572v1…