PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Learning and Reasoning over Model Slices

Yuheng Wang, Runde Yang, Zhuohan Bao, Qiran Zhang, Huatao Li, Yiyang Zheng, Lin Wu, Jingru Fan, Shu Yao, Yihan Li, Zhi Li, Chen Qian

arXiv:2610.12927v1cs.AI

TL;DR

Single-model reasoning can leave intermediate errors undetected, and prior reasoning may be difficult to reassess independently. The paper trains heterogeneous models to collaborate through slicing and pyramid supervision; across four task categories, they match or outperform the large model and multi-agent baselines while showing higher inference efficiency.

  • Problem

    When the same model generates and evaluates a solution, self-bias may limit error detection, allowing intermediate errors to influence later reasoning.

    Source: §1 · p. 1

  • Method

    Collaborative learning constructs smaller heterogeneous models under a comparable parameter budget and uses pyramid supervision to guide their reasoning and interaction decisions.

    Source: §1 · pp. 1–2

  • Results

    Across four task categories, trained heterogeneous models match or outperform the original model and MAS/MARL baselines, with higher inference efficiency than those baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support heterogeneous models as complementary reasoning collaborators that reassess prior reasoning across distinct parameterizations.

    Source: §5 · p. 9

    Generalization to larger model pools, longer interaction horizons, and unseen task distributions remains unexplored.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) excel across diverse tasks, benefiting from knowledge priors, in-context reasoning, and test-time computation. Despite these strengths, single-model reasoning remains vulnerable to error propagation, as later steps depend on the model's earlier outputs, leaving reasoning blind spots difficult to independently reassess. To address this limitation, we develop \textit{collaborative learning}, encompassing the construction of heterogeneous collaborators and the learning of effective cross-model collaboration. Specifically, \textit{model slicing} replaces a large model with smaller, heterogeneous models under a comparable parameter budget, enabling prior reasoning to be reassessed across distinct model parameterizations. \textit{Pyramid supervision} combines protocol distillation with step-level supervision to refine reasoning and interaction decisions throughout collaboration. We evaluate our approach across four task categories: multi-hop question answering, mathematical reasoning, code generation, and open-ended constrained generation. The trained heterogeneous models achieve comparable or better task performance than the original large model and established multi-agent baselines, including reinforcement-learning-based methods, while also achieving higher inference efficiency than the multi-agent baselines and exhibiting differentiated collaboration patterns across tasks. These findings show that learned collaboration turns model heterogeneity into complementary reasoning capabilities.

1 INTRODUCTION

The introduction presents collaborative learning as a way for heterogeneous models to reassess prior reasoning and learn effective cross-model collaboration. Across four task categories, the trained models match or outperform the original model and multi-agent baselines while showing efficient, differentiated collaboration.

1 INTRODUCTION

  • Single-model reasoning may leave intermediate errors undetected because the same model generates and evaluates a solution, motivating independent reassessment across model parameterizations.The introduction notes that errors in intermediate steps may influence subsequent reasoning.Source: §1 · p. 1
  • Model slicing replaces one large model with smaller heterogeneous models under a comparable parameter budget, using task-state handoffs to reassess prior reasoning.This provides a structural basis for models to question, extend, or revise prior conclusions.Source: §1 · pp. 1–2
  • Pyramid supervision combines interaction initialization with fine-grained guidance over reasoning and interaction decisions, alongside task-level rewards.The framework addresses the limited guidance outcome-level supervision provides for intermediate decisions.Source: §1 · p. 2
  • Across four task categories, trained heterogeneous models match or outperform the original model and MAS/MARL baselines, with higher inference efficiency than those baselines.Their interaction trajectories also show reduced reasoning redundancy and differentiated collaboration patterns.Source: §1 · p. 2
  • The proposed trainable paradigm uses interaction trajectories to support independent reassessment and progressive learning of cross-model collaboration.The approach is evaluated on multi-hop question answering, mathematical reasoning, code generation, and open-ended constrained generation.Source: §1 · p. 2

2 METHOD

Collaborative reasoning lets heterogeneous models successively contribute to a solution through explicit task-state handoffs. Collaborative learning builds these collaborators through model slicing and trains their interactions through pyramid supervision.

2.1 COLLABORATIVE REASONING

  • Collaborative reasoning uses a shared protocol for successive model contributions, with control retained, handed off, or terminated at each step.Each step includes the active model, task input and interaction history, current reasoning, and interaction action.Source: §2.1 · p. 3

2.2 COLLABORATIVE LEARNING

  • Collaborative learning combines heterogeneous collaborator construction under a comparable parameter budget with supervision that refines reasoning and interaction decisions.It integrates task-level rewards throughout collaborative reasoning.Source: §2.2 · pp. 3–4

2.2.1 MODEL SLICING

  • Model slicing replaces a large model with smaller, independently parameterized models whose combined parameter count is comparable to the original.The models are not direct partitions of the large model’s weights; task-state handoffs support reasoning-state isolation across their distinct parameterizations.Source: §2.2.1 · p. 4

2.2.2 PYRAMID SUPERVISION

  • Protocol distillation initializes cross-model interaction using teacher-generated trajectories, with the largest model slice serving as the teacher.The teacher system follows the target models’ interaction protocol to generate multi-step trajectories, and qualified trajectories are retained for distillation.Source: §2.2.2 · p. 4
  • Step-level supervision evaluates reasoning and interaction decisions in complete trajectories, then integrates those scores with the final task reward.The original large model evaluates each step in trajectory context; α controls the trade-off between step-level and task-level rewards.Source: §2.2.2 · p. 5
  • Models optimize their own active steps using combined rewards, increasing positive-reward responses and decreasing negative-reward responses.Rewards reflect both step evaluation conditioned on the complete trajectory and the final task outcome, allowing differentiated collaboration patterns to emerge through training.Source: §2.2.2 · p. 5

3 EXPERIMENTS

Across five benchmarks in four task categories, the experiments compare a parameter-matched heterogeneous collaborative system with single-model, multi-agent, and reinforcement-learning baselines. The results report performance gains, contributions from both training mechanisms, more efficient inference, and task-dependent interaction patterns.

3.1 EXPERIMENTAL SETUP

  • The evaluation covers five benchmarks across multi-hop QA, mathematics, code generation, and open-ended constrained generation, against single-agent, multi-agent, and MARL baselines.The reported metrics include accuracy and F1, pass@1 variants, Coverage, and Explicit.Source: §3.1 · pp. 5–6

3.2 DOES COLLABORATIVE LEARNING LEAD TO IMPROVED PERFORMANCE?

  • With 13.7B parameters, the method gains 2.38% on average over Qwen3-14B and 2.08%–6.87% over established MAS/MARL baselines across ten metrics.It also beats a 24B homogeneous MAS on all ten metrics and exceeds same-procedure Homo. CL by an average absolute 6.02%.Source: Table 1 · p. 6, §3.2 · pp. 6–7

3.3 HOW MUCH DOES EACH MECHANISM CONTRIBUTE TO PERFORMANCE?

  • Removing collaborative training, protocol distillation, and step-level scoring lowers ten-metric average scores by 9.37, 8.85, and 5.41 points, respectively; every variant trails the full method.The ablation results indicate that heterogeneity alone is insufficient for the full gains, while both supervision components contribute to learned collaboration.Source: §3.3 · p. 7
  • Without protocol distillation, MuSiQue corrections fell from 594 to 171 as first-answer EM rose from 21.02% to 25.24%; without step-level scoring, trajectories lengthened from 3.4 to 3.8 steps as corrections fell 18.5%.Protocol-following failures also increased on MATH, and constraint-satisfying Conifer trajectories more often failed by termination without protocol distillation.Source: §3.3 · p. 7

3.4 HOW EFFICIENT AND EFFECTIVE IS COLLABORATIVE REASONING?

  • Reasoning overlap on MuSiQue fell from 48.3%, 24.1%, and 55.2% to 0.2%, 0.3%, and 0.5%, while correct-answer retention ranged from 80.42% to 98.26% across four tasks.Retention was 80.42% on MuSiQue, 97.94% on GSM-Hard, 98.26% on MultiPL-E, and 98.00% on MATH; Conifer was excluded because it lacks an intermediate-candidate correctness criterion.Source: §3.4 · p. 8, §3.5 · p. 8
  • The system uses 39.1%–84.8% fewer invocations than four multi-agent baselines and 46.7%–86.0% fewer than MARL baselines across five benchmarks.It also has the lowest parameter-weighted output-token count on four of five benchmarks and reduces that count by 8.2%–89.4% versus MARL baselines.Source: §3.4 · pp. 7–8, Figure 4 · p. 8

3.5 WHAT COLLABORATION PATTERNS EMERGE FROM COLLABORATIVE LEARNING?

  • Long linear trajectories rose from 9.18% to 60.12% on MuSiQue but fell from 10.95% to 8.58% on MultiPL-E; short cyclic lengths shifted differently on GSM-Hard and MATH.The patterns indicate that training does not uniformly increase interaction complexity or length.Source: Figure 5 · p. 8, §3.5 · p. 8
  • On MuSiQue, Qwen3-4B’s handoff rate rose from 48% to 96% as its termination rate fell from 50% to 4%, while Qwen3-8B’s final-answer share rose from 13.0% to 58.4%.The division of labor differed on GSM-Hard, where Qwen3-4B’s final-answer share rose from 34.09% to 64.39%.Source: §3.5 · p. 9

4 RELATED WORK

Prior reasoning research spans single-model prompting and training, multi-model composition, and collaborative interaction. The paper distinguishes learned collaboration among heterogeneous models from composition systems and approaches with prescribed roles.

Reasoning in LLMs

  • Single-model prompting and training methods remain bounded by one model’s knowledge and reasoning biases, so blind spots and intermediate errors can persist (Xie et al., 2026).The reviewed methods include chain-of-thought prompting (Wei et al., 2022), exploration and revision, bootstrapped demonstrations, distillation, and outcome- or process-level supervision.Source: §4 · p. 9

From Single-Model Reasoning to Multi-Model Composition

  • Multi-model composition selects models, distributes tasks, sequences invocations, routes extra models, or aggregates outputs, but does not necessarily entail joint solution development (Zhang et al., 2026b).The passage cites examples across these approaches, including model selection and task distribution (Agashe et al., 2025).Source: §4 · p. 9

Multi-Model Collaborative Reasoning

  • Collaborative reasoning studies how models jointly develop solutions, but representative systems often prescribe roles; this paper instead trains heterogeneous small models without predefined roles (Zhao et al., 2026b).The stated total parameter budget is comparable to that of one large model.Source: §4 · p. 9

5 CONCLUSION

The framework combines model slicing and pyramid supervision to support collaboration across heterogeneous models. Across four task categories, these models match or outperform large-model and multi-agent baselines, while the study leaves several forms of generalization unexplored.

5 CONCLUSION

  • Across four task categories, trained heterogeneous models match or outperform the large model and MAS and MARL baselines while requiring fewer model invocations than multi-agent frameworks.The comparison uses a total parameter budget comparable to that of the original large model.Source: §5 · p. 9
  • Model slicing enables reasoning-state isolation through task-state handoffs, while pyramid supervision combines protocol distillation with step-level supervision over model-generated trajectories.Source: §5 · p. 9
  • Interaction analysis reveals reduced reasoning redundancy, stronger correct-answer retention, and task-dependent interaction topologies and functional specialization.Source: §5 · p. 9
  • Generalization to larger model pools, longer interaction horizons, and unseen task distributions remains unexplored; computational cost may also exceed parameter-weighted output-token estimates due to latency and memory overhead.Pyramid supervision also depends on inspector and evaluator quality, whose biases may affect step-level feedback.Source: §5 · p. 9
Loading 2610.12927v1…