PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Yunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu, Tianjin Huang, Yuanyuan Shi, Ziang Xiao, Nuno Vasconcelos, Yijiang Li

arXiv:2608.17253v1cs.LGcs.AIcs.CV

TL;DR

Ground-truth rewards are costly and scarce for advanced reasoning, while self-rewarding RL can amplify errors and collapse. Co-RL trains decoupled, diverse agents using peer-derived rewards and improves text-only and multimodal reasoning without ground-truth labels.

  • Problem

    Ground-truth supervision for reasoning RL is costly and scarce, while self-rewarding can amplify correlated errors, reduce diversity, and cause training collapse.

  • Method

    Co-RL jointly optimizes decoupled, parameter-independent agents using rewards derived from diverse peers’ pseudo-answers rather than ground-truth labels.

  • Results

    CO-RL consistently improves text-only and multimodal reasoning, outperforming prior label-free approaches while matching or surpassing supervised training in several settings.

  • Takeaways & Limitations

    Peer-derived supervision can provide an effective label-free learning signal while preserving diversity across language and vision-language reasoning tasks.

  • Takeaways & Limitations

    Self-rewarding approaches rely exclusively on a single model’s view, which can reinforce biases and errors and lead to substantial training collapse.

Abstract

from arXiv · show

Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.

1 Introduction

Co-RL addresses the limits of ground-truth and self-generated supervision by training decoupled agents with peer-derived rewards. Diverse cohorts reduce correlated errors, enabling label-free reasoning improvements across text-only and multimodal settings without ground-truth labels.

  • Ground-truth supervision strengthens RL reasoning but is costly and increasingly scarce as target capabilities approach or exceed human evaluation.
  • Self-reinforcement can amplify biases and suboptimal behaviors, reduce response diversity, and produce homogeneous outputs or training collapse.
  • Co-RL trains multiple parameter-independent models simultaneously, using each agent’s completions and majority-voted pseudo-answers to generate peer-derived rewards.
  • Cohort diversity across architectures, pretrained weights, model sizes, and input formulations exposes agents to distinct inductive biases and reduces correlated errors.
  • 3.0–8.6% average gains improve four LLMs across seven text-only benchmarks, while Co-RL also improves across multimodal domains without ground-truth labels.

2 Related Work

Prior work improves reasoning through self-rewarding RL, cross-view supervision, multi-agent interaction, and multimodal RLVR. However, single-model self-rewarding can amplify biases and errors, while similar views may reinforce shared mistakes.

  • Self-rewarding RL: Self-rewarding RL reduces reliance on ground-truth labels through model-generated rewards, self-correction, and self-generated curricula.These approaches include self-rewarding mechanisms, self-correction based on a model’s own judgments, and zero-data self-evolution.
  • Self-rewarding RL: Single-model self-rewarding can amplify existing biases and errors without an external corrective signal, often causing substantial training collapse.
  • Co-training, cross-view supervision, and its origins: Co-training uses multiple views to supervise one another, but highly similar views tend to reinforce the same errors.Co-training originated from the idea that two conditionally independent views can teach each other.
  • Multi-agent RL: Multi-agent reasoning methods enable models to critique, revise, debate, reach consensus, or operate through orchestrated pipelines.A newer line trains interacting agents, including methods that convert LLM-judge scores into rewards.
  • RLVR for multimodal reasoning: Multimodal RLVR applies R1-style rule-based rewards to vision-language reasoning and improves reward design and training stability through curriculum or perception-aware rewards.

3 Preliminary

The preliminary section formalizes response-level RL and GRPO, then reviews label-free reward construction from model outputs. It highlights that self-generated rewards can reinforce bias, reduce diversity, and cause training collapse, motivating independent cohort rewards.

  • RL formulation: RL trains an autoregressive policy to maximize scalar rewards evaluating generated responses, which typically include reasoning trajectories and final answers.The objective is J(θ) = E_x∼D, y∼πθ(·|x)[r(x, y)].
  • GRPO: GRPO samples K responses, normalizes their rewards into group-relative advantages, and optimizes a clipped surrogate objective with KL regularization.It avoids requiring a separately trained critic model, while δ controls clipping and β controls KL regularization strength.
  • Label-free rewards: Label-free methods derive scalar rewards from model outputs using majority-vote agreement, token-level confidence, or predictive-distribution statistics.TTRL uses majority-vote agreement; Intuitor and RENT use token-level predictive distributions, after which GRPO converts rewards into group-relative advantages.
  • Limitations: Self-generated signals can reinforce biases and suboptimal behaviors, reduce response diversity, and produce homogenization or training collapse.The section reports that prolonged TTRL training degenerates and collapses, as shown in Figure 2(b) and (c).
  • Cohort rewards: Co-RL addresses this limitation by deriving rewards directly from votes by independent cohort models rather than from the policy’s own view.For more than two agents, votes pass along a directed ring.

4 Method

Co-RL is a label-free multi-agent RL framework in which decoupled agents supervise one another using peer-generated majority-vote rewards, without shared parameters or gradients. It further promotes corrective learning by diversifying optimization, model families and sizes, and input formulations to reduce correlated errors.

  • Core framework: Co-RL trains multiple decoupled agents on unlabeled problems, coupling their optimization only through peer-provided majority-vote pseudo rewards.Agents share neither parameters nor gradients and independently generate completions before supervising one another.
  • Cross-agent rewards: Each agent is rewarded 1 when its extracted answer matches its designated peer’s majority-vote pseudo-label, and 0 otherwise.Peer supervision is cyclic, excludes the supervised agent’s own answers, and becomes mutual in the two-agent setting.
  • Policy optimization: All agents generate rollouts and compute pseudo-labels before independent policy updates using group-relative advantages in GRPO.The framework updates every agent at each optimization step and is compatible with other sequence-level reward policy optimizers.
  • Cohort diversity: CO-RL increases diversity through independently maintained policies, heterogeneous pretrained model families, and different model sizes, which provide distinct inductive biases and error profiles.Independent updates prevent direct coupling, while pairing different families or capacities enables more informative cross-agent reward signals.
  • Input formation: Input diversity rewrites each MATH problem for one agent while preserving its answer and sample order, typically changing the concrete scenario rather than merely substituting words.One agent trains on the original prompt and the other on the DeepSeek-V3 rewrite.

5 Theoretical Analysis

The theoretical analysis contrasts self-rewarding’s self-confirming updates with Co-RL’s cross-agent supervision. Co-RL expands the basin of correct convergence and can exploit complementary agent expertise to correct errors that self-rewarding reinforces.

  • Training dynamics: Self-rewarding uses pseudo-labels derived from the same agent’s predictions, so systematic errors can be reinforced rather than corrected.Co-RL instead derives supervision from other agents, decoupling the supervision signal from the agent being optimized.
  • Self-rewarding dynamics: Self-rewarding amplifies the currently favored answer regardless of correctness, suppressing the correct answer when p < 1/2.This behavior is characterized as self-confirming dynamics under the two-outcome answer model.
  • Co-RL convergence: Under symmetric two-agent Co-RL dynamics, (1, 1) and (0, 0) are asymptotically stable, while (1/2, 1/2) is a saddle with separatrix pA + pB = 1.The theorem characterizes the basin of attraction for correct versus incorrect consensus from interior initialization.
  • Co-RL convergence: If pA(0) + pB(0) < 1, the two agents converge to incorrect consensus, (pA(t), pB(t)) → (0, 0).This is the theorem’s stated incorrect-convergence condition.
  • Complementary expertise: With complementary accuracies of 0.55 each, self-rewarding reaches final accuracy 0.5, whereas Co-RL reaches accuracy 1.0 because pA + pB = 1.1 on every prompt.The example uses (pA, pB) = (0.9, 0.2) on half the prompts and (0.2, 0.9) on the other half.

6 Experiments

Experiments evaluate Co-RL across language and vision-language reasoning using diverse models, datasets, and benchmarks. Co-RL consistently improves over label-free and multi-agent baselines, including in three-agent and multimodal settings.

  • Experimental setup: Experiments train language models on level 3–5 MATH and vision-language models on MMR1-Math and multimodal-open-r1.The second multimodal dataset tests whether gains depend on the baseline’s training data.
  • Experimental setup: The study evaluates Qwen, Llama, Qwen2.5-VL, InternVL3.5, and Gemma-3 models, pairing vision-language models at comparable scales.Language experiments include both same-family and cross-family pairings, while multimodal experiments span multiple visual and language architectures.
  • Baselines and evaluation: Language experiments compare Co-RL with TTRL, Intuitor, RENT, Co-Rewarding-II, and supervised GRPO with ground-truth rewards across seven reasoning benchmarks.The benchmarks cover mathematics, code generation, and science.
  • Language-model results: Co-RL outperforms all self-rewarding language-model methods, with Different family+ achieving the strongest label-free average performance for both model families.The framework compares same-family agents, different-family agents, and different-family agents with decoupled training data.
  • Three-agent scaling: 7.8%, 6.0%, and 8.2% average gains improve Qwen2.5-3B, Llama-3.2-3B-Instruct, and Qwen3-1.7B, respectively, in three-agent CO-RL.These gains are reported against independently trained versions using ground-truth or self-generated majority-vote rewards.
  • Vision-language results: Co-RL achieves the best average performance in three of four 2B–3B multimodal settings and remains competitive with ground-truth supervision at larger scales.At larger scales, it consistently outperforms TTRL and surpasses GT-Reward for Gemma-3-12B.

7 Ablation Study

Co-RL exhibits more stable training dynamics than self-rewarding baselines and remains effective when training and inference budgets are matched. Across text and multimodal settings, it achieves the best average score under the matched-budget comparison.

  • Training dynamics and stability: Co-RL maintains stable reward variation and completion lengths across text models, while self-rewarding baselines can suffer reward collapse, length degeneration, or divergence.Figure 4 evaluates validation accuracy, normalized within-group reward standard deviation, and mean completion length across four backbone scales.
  • Training dynamics and stability: For VLMs, agents retain partial agreement while the accuracy of exchanged pseudo-labels improves throughout training.
  • Controlling for training and inference budgets: CO-RL achieves the best average score across text and multimodal settings when compared with a self-rewarding baseline matched for two-agent training and inference budgets.The comparison trains two base models independently with TTRL and ensembles four rollouts per model for majority voting at inference.
  • Vision-language ablation: The small-pair vision-language ablation uses Qwen2.5-VL-3B and InternVL3.5-2B trained separately on open-r1 and MMR1.Base and GT-Reward are reference conditions excluded from ranking; Base is identical across training sets under the corrected multiple-choice grader.

8 Conclusion

CO-RL is a label-free multi-agent RL framework in which agents learn from peer-derived rewards rather than ground-truth labels or external judges. It improves reasoning across text-only and multimodal benchmarks, while cross-agent supervision theoretically enables correction of errors that self-rewarding RL would reinforce.

  • Framework: CO-RL trains multiple agents using rewards constructed from peers’ predictions instead of ground-truth labels or external judges.The framework is designed for reasoning tasks across language and vision-language models.
  • Empirical findings: CO-RL consistently improves diverse LLMs and VLMs across text-only and multimodal reasoning benchmarks, outperforming prior self-rewarding and multi-agent RL approaches.
  • Analysis and future work: Cross-agent supervision expands the initial conditions that converge to correct solutions, helping correct errors that self-rewarding RL would otherwise reinforce.Future work should examine agent number, diversity, interaction topology, and adaptive supervision mechanisms for exploiting complementary expertise.

A Pre-training Error Decoupling · B Complete Proof · B.1 Proof of Proposition 1

Co-RL benefits from genuinely decoupled peer errors: different-family models substantially reduce error overlap and increase complementarity, unlike resampling or scaling within one lineage. The proof formalizes how fixed pseudo-labels induce GRPO dynamics, recovering the self-rewarding, cross-agent, and symmetric two-agent updates.

  • A Pre-training Error Decoupling: Different-family pairs achieve κ ≤0.42 and c ≥29.4, whereas same-family and seed-only pairs remain at κ ≥0.51 and c ≤24.6.No pair falls between these groups at either model scale.
  • A Pre-training Error Decoupling: Crossing model families lowers average κ from 0.53 to 0.38 and raises complementarity c from 23.2 to 30.8.Changing size, generation, or sampling seed within one lineage does not distinguish same-family from seed-only groups.
  • A Pre-training Error Decoupling: With either anchor fixed, same-family partners lower κ by 0.04 or 0.07, while different-family partners lower it by 0.18 or 0.16 and raise c by 9.2% or 10.4%.Capability is identical along each seed-only ladder, isolating partner source as the variable.
  • A Pre-training Error Decoupling: Different-family cohorts provide peer targets based on different failures, while same-family cohorts repeat the agent’s likely answer on three quarters of problems.The analysis evaluates pretrained checkpoints before RL on 500 MATH problems using zero-shot, single-sample evaluation at T=0.8.
  • B Complete Proof: The proof derives a common GRPO update for a fixed pseudo-label, then specializes it to self-rewarding and cross-agent supervision.The analysis uses log-odds dynamics in the infinitesimal-update regime, where clipping is locally inactive.
  • B.1 Proof of Proposition 1: For a rollout group with 0 < C < K correct responses, the proof derives normalized group-relative advantages; when C ∈{0, K}, the centered update is zero.C denotes the number of correct responses among K rollouts.
  • B.1 Proof of Proposition 1: For two symmetric agents using the same learning rate η, the resulting coupled dynamics are ˙pA = qK(pA)ϕK(pB) and ˙pB = qK(pB)ϕK(pA).Each agent’s pseudo-label is the majority vote of the other agent’s rollouts.

B.2 Proof of Proposition 2 · B.3 Proof of Theorem 1

The proofs show that majority-vote self-rewarding creates self-confirming dynamics: probabilities below or above 1/2 converge toward 0 or 1, while 1/2 is unstable. For two-agent Co-RL, a conserved quantity yields the separatrix pA + pB = 1, stable consensus equilibria, and a strictly larger correct-convergence basin than independent self-rewarding.

  • B.2 Proof of Proposition 2: Majority-vote self-rewarding reinforces the currently favored answer and induces self-confirming dynamics.The proof establishes that the update terms share a common sign away from the midpoint.
  • B.2 Proof of Proposition 2: If 0 < p(0) < 1/2, then p(t) monotonically decreases and converges to 0.The trajectory is bounded below, and continuity rules out an interior limiting value.
  • B.2 Proof of Proposition 2: If 1/2 < p(0) < 1, then p(t) monotonically increases and converges to 1, making p = 1/2 an unstable equilibrium.The increasing trajectory is bounded above, while the same argument excludes interior limits.
  • B.3 Proof of Theorem 1: The two-agent dynamics preserve FK(pA) − FK(pB) = constant along every interior trajectory.FK is strictly decreasing on (0, 1/2), strictly increasing on (1/2, 1), and satisfies FK(1/2) = 0.
  • B.3 Proof of Theorem 1: When pA + pB < 1, agent B crosses below 1/2 first, and the trajectory converges to (0, 0).The case pB < 1/2 < pA follows symmetrically.
  • B.3 Proof of Theorem 1: The line pA + pB = 1 is invariant and forms the interior separatrix, with trajectories converging to (1/2, 1/2).The equilibria (1, 1) and (0, 0) are asymptotically stable, whereas (1/2, 1/2) is a saddle whose stable manifold is this invariant line.
  • B.3 Proof of Theorem 1: Co-RL has the basin pA + pB > 1 and therefore a strictly larger basin of correct convergence than independent self-rewarding.The proof explicitly contrasts the Co-RL basin with the independent self-rewarding basin.

C Rephrased Training Questions · D Complete Experiment Results · D.1 Results on 7B and 8B language models

Co-RL uses answer-preserving, substantially rephrased questions to decouple agents’ training data, and its benefits persist at 7B–8B scale. Different-family agents achieve the strongest reported label-free results, including surpassing ground-truth reward on one model.

  • C Rephrased Training Questions: Rephrased questions preserve both answers and row alignment while transforming many problems into concrete scenarios and roughly doubling their length.In a sample of 300 pairs, every rewrite preserved the answer and alignment.
  • C Rephrased Training Questions: DeepSeek-V3 generates the rephrased copy used to train the second agent, while the first agent trains on the original MATH questions.The two agents are data-decoupled and share aligned question rows.
  • D.1 Results on 7B and 8B language models: At larger scale, the comparison pairs Qwen2.5-7B with Llama-3.1-8B-Instruct across the same seven benchmarks.The reported trends from 3B models continue at this scale.
  • D.1 Results on 7B and 8B language models: All CO-RL variants improve over their respective base models on average, with different model families generally producing stronger gains than the same family.Further decoupling the training data is also discussed as part of the larger-scale comparison.
  • D.1 Results on 7B and 8B language models: 0.8% on Qwen2.5-7B and 1.1% on Llama-3.1-8B-Instruct are CO-RL (Different family+) gains over the strongest self-rewarding baseline.This variant achieves the best average performance for both models among prior label-free methods.
  • D.1 Results on 7B and 8B language models: 47.7 vs. 47.1 is the Llama-3.1-8B-Instruct comparison between CO-RL (Different family+) and GT-Reward without ground-truth labels.The result supports the persistence of cross-agent supervision benefits as model scale increases.

D.2 Results on 7B–12B Vision-Language Models … D.5 Evaluation Details

CO-RL improves larger vision-language models across heterogeneous architectures, remains stable during training, and benefits from cross-agent supervision beyond simply training or ensembling two models. Evaluation uses task-specific decoding and grading protocols, with released engineering fixes preserving training and evaluation semantics.

  • D.2 Results on 7B–12B Vision-Language Models: 7.2%, 6.3%, and 5.8% average gains over Qwen2.5-VL-7B, InternVL3.5-8B, and Gemma-3-12B base models persist across four multimodal benchmarks.InternVL3.5-8B serves as the shared training partner.
  • D.2 Results on 7B–12B Vision-Language Models: 47.56% vs. 45.17% shows CO-RL surpassing GT-Reward on Gemma-3-12B, while approaching GT-Reward on Qwen2.5-VL-7B and InternVL3.5-8B.The gains extend across substantially different visual encoders, language backbones, model scales, and heterogeneous vision-language architectures.
  • D.3 Training Dynamics and Stability: CO-RL steadily improves validation accuracy while maintaining non-degenerate reward standard deviation and relatively stable completion lengths across four text-model families.Several self-rewarding methods instead become unstable during training; RENT rapidly drives reward standard deviation toward zero and often sharply increases completion length.
  • D.3 Training Dynamics and Stability: CO-RL improves multimodal evaluation accuracy while holding completion length stable, whereas TTRL eventually degrades in both accuracy and response length.Agent agreement remains below full agreement while exchanged pseudo-label accuracy steadily increases, indicating that the agents do not simply converge to identical behavior.
  • D.4 Controlling for the Two-Agent Training Budget: CO-RL ensembles consistently achieve the best macro-average on text benchmarks and both multimodal settings, unlike matched-budget TTRL ensembles, which provide only limited gains.The matched baseline independently trains the same two base models and pools four responses from each for majority voting, showing the advantage is not merely training two models.
  • D.5 Evaluation Details: Language evaluation uses temperature 0.6, top-p 0.95, and a 3072-token generation budget; AMC uses avg@8, while other benchmarks use one sample.GPQA uses the Diamond subset with boxed-answer prompts, and rule-based graders apply math-aware normalization while accepting option letters and values.
  • D.5 Evaluation Details: Vision-language evaluation uses greedy decoding with a 16384-token budget, 1024-pixel maximum long-side image resizing, and benchmark-specific test splits.The evaluated sets include MM-UPT, MathVision, MathVerse, MathVista, and We-Math, with trained checkpoints prompted in their training format.
  • D.5 Evaluation Details: Released engineering fixes repair crashes or backend choices without changing training or evaluation semantics, including Gemma-3 tokenization and Qwen2.5-VL attention handling.Gemma-3 additionally uses token-level importance-ratio truncation because its vLLM and training forward passes exhibit about 0.13 per-token log-probability drift.
Loading 2608.17253v1…