PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say

Jacob Fein-Ashley, Dhruv Parikh, Rajgopal Kannan, Viktor Prasanna

arXiv:2509.21164v1cs.LG

TL;DR

Prior multi-LLM approaches route experts, aggregate outputs through iterative exchanges, or fuse weights, leaving latent-level collaboration across heterogeneous models unaddressed. MoT routes each query to selected experts and lets a primary expert integrate peers’ hidden states in a shared latent space. It surpasses AVENGERS on both ID and OOD benchmarks with single-pass inference.

  • Problem

    Prior multi-LLM approaches route queries, aggregate outputs through costly multi-turn exchanges, or fuse weights that typically require architectural homogeneity, rather than collaborating in latent space.

    Source: §1 · p. 2

  • Method

    MoT selects top-K heterogeneous experts per query and uses shared-latent interaction layers for primary-expert cross-attention over peers’ hidden states, training only the router and interaction layers.

    Source: §1 · p. 2

  • Results

    MoT exceeds AVENGERS by +0.38% (ID) and +2.92% (OOD) across five ID and three OOD benchmarks, with comparable wall-clock cost.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MoT combines per-query collaboration among heterogeneous frozen experts with single-pass inference and runtime comparable to routing baselines.

    Source: §1 · p. 2

    Experiments use a modest pool of frozen experts with limited parameter counts and interaction depth; systematic scaling remains future work.

    Source: K LIMITATIONS · p. 22

Abstract

from arXiv · show

Open-source Large Language Models (LLMs) increasingly specialize by domain (e.g., math, code, general reasoning), motivating systems that leverage complementary strengths across models. Prior multi-LLM approaches either (i) route a query to one or a few experts and generate independently, (ii) aggregate outputs from each model via costly multi-turn exchanges, or (iii) fuse weights into a single model-typically requiring architectural homogeneity. We introduce Mixture of Thoughts (MoT), a simple method for latent-level collaboration among heterogeneous experts under a global routing scheme. For each query, a lightweight router selects top-$K$ experts and designates a primary expert; uniformly placed interaction layers project hidden states into a shared latent space where the primary expert performs cross-attention over its active (selected) peers. Pre-trained experts remain frozen; only the router and the lightweight interaction layers are trained with a novel joint training objective that improves both the expert selection and inter-expert collaboration. Across five in-distribution (ID) and three out-of-distribution (OOD) benchmarks, MoT surpasses the current routing and aggregation-based state-of-the-art, Avengers, by $+0.38\%$ and $+2.92\%$, respectively. Further, MoT significantly outperforms the best-performing single model. It achieves this with single-pass inference, runtime comparable to routing baselines, and none of the overheads of iterative aggregation. MoT offers a simple latent-space mechanism for combining heterogeneous LLMs, a practical step toward broader multi-LLM collaboration. Our code is publicly available at https://github.com/jacobfa/mot.

1 INTRODUCTION

Specialized open-source LLMs offer complementary skills, but prior approaches route queries, aggregate outputs, or fuse weights without fine-grained latent-space collaboration. MoT combines global routing with hidden-state exchange among heterogeneous experts, and reports gains over prior multi-LLM methods on ID and OOD benchmarks.

1 INTRODUCTION

  • Prior approaches route to experts, aggregate responses, or fuse parameters; these coarse-grained or compute-heavy strategies can weaken generalization and may require homogeneous architectures.The introduction identifies a missing form of fine-grained latent-space interaction; Figure 2 contrasts routing and output aggregation with MoT.Source: §1 · p. 2, Figure 2 · p. 2
  • MoT routes each query to top-K heterogeneous experts, then integrates their projected hidden states into a primary expert through cross-attention while keeping backbones frozen.A joint objective trains the router and interaction layers; inference uses one forward pass through selected experts.Source: §1 · p. 2
  • Unlike output-level aggregation, MoT lets experts integrate hidden representations, while preserving per-query adaptive collaboration among frozen, heterogeneous experts.The contributions also report robustness to expert count, expert dropout, interaction depth and placement, and shared latent size.Source: §1 · pp. 2–3
  • MoT exceeds AVENGERS by +0.38% (ID) and +2.92% (OOD), while improving over the strongest single model by +10.95% (ID) and +9.03% (OOD) at comparable wall-clock cost.The evaluation covers language understanding, code generation, and mathematical reasoning; MoT takes 51.8 minutes versus AVENGERS’ 51.3 minutes.Source: §1 · p. 2

2 RELATED WORK

Prior multi-LLM work routes queries to selected experts, coordinates models through iterative response-level exchanges, or fuses model weights. MoT instead enables latent-space collaboration across heterogeneous experts in a single pass.

2 RELATED WORK

  • Routing methods select experts but typically fuse only final generations; response-level systems collaborate through costly multi-round exchanges, while weight fusion often assumes homogeneous architectures and removes per-query adaptivity.MoT instead keeps heterogeneous backbones frozen and lets the primary expert aggregate token-level information from selected peers in a single pass.Source: §2 · p. 3

3 METHOD

MoT combines sparse global routing with shared-latent interaction layers so a primary expert can integrate representations from heterogeneous, frozen peers. The method trains the router and interaction components, then generates in a single pass.

3.1 OVERVIEW

  • Experts of differing depths are divided into Q stacks, creating Q interaction stages where routed experts project states into a shared space for primary-expert cross-attention.The expert backbones stay frozen; only the router and projection/attention layers are trained.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 GLOBAL ROUTING

  • For each prompt, a lightweight router scores all experts, activates the top-K, and designates the highest-scoring expert as the primary decoder.The primary expert alone generates final output tokens, integrating information from its active peers.Source: §3.2 · p. 4, Figure 4 · p. 5

3.3 INTERACTION LAYERS

  • At each stack boundary, active experts project hidden states into a shared dimension, and the primary expert uses cross-attention to integrate those states.The interaction layers use multi-head attention with a block-causal mask for autoregressive visibility across active experts’ tokens.Source: §3.3 · pp. 5–6, Figure 5 · p. 5
  • Non-primary experts skip cross-attention and pass through projector pairs, while continuing to decode so their latent trajectories can inform later interactions.Only the primary expert executes cross-attention; all experts train lightweight projectors.Source: §3.3 · p. 6

3.4 TRAINING

  • Training combines autoregressive language-modeling loss with entropy and load-balancing regularizers, and uses a routing-consistency term to stabilize independently perturbed Top-K selections.A straight-through estimator carries gradients through soft Top-K scores while hard indices route experts in the forward pass.Source: §3.4 · p. 6

3.5 INFERENCE

  • Inference routes the prompt to top-K experts, runs their stacks in parallel, and has the primary expert generate tokens while repeatedly integrating peers’ projected states.Non-primary outputs are masked if an expert halts early; key–value pairs can be cached across experts.Source: §3.5 · p. 6

4 EXPERIMENTS

MoT combines a global router with latent interaction among selected experts, and outperforms prior routing and aggregation baselines across in-distribution and out-of-distribution evaluations. Ablations examine the interaction layers, model size, training losses, and resilience to fewer available experts.

4.1 SETUP

  • Experiments use seven 7B–8B experts; training combines five ID benchmark train splits, while three unseen datasets assess OOD generalization.Baselines include training-free, clustering-based, and supervised routers, evaluated using the same expert pool and datasets.Source: §4.1 · pp. 6–7

4.2 RESULTS

  • MoT beats Avengers by +0.38% on ID (60.53 vs. 60.30) and +2.92% on OOD (47.92 vs. 46.56).It leads on four of five ID tasks and all three OOD tasks; ID runtime is 51.8 vs. 51.3 minutes.Source: Table 1 · p. 7, §4.2 · p. 7, Table 2 · p. 8

4.3 ABLATIONS

  • Increasing interaction layers from Q=2 to Q=8 raises ID average from 56.50 to 60.53 and OOD average from 44.20 to 47.92, with diminishing gains beyond Q=6.More layers add computation and parameters; Figure 6a shows the trend.Source: §4.3 · p. 7, Figure 6 · p. 9
  • Deep placement yields 58.65 ID and 46.23 OOD averages, while uniform placement remains competitive at 58.37 and 46.00.Figure 6b compares shallow, intermediate, deep, and uniformly spaced Q=4 layers.Source: §4.3 · p. 7, Figure 6 · p. 9
  • A smaller MoT reduces inference time by ∼32% with a modest accuracy drop, while removing primary-expert cross-attention lowers ID and OOD accuracy.The reduced configuration uses Q=6, shared dimension 1024, and router hidden size 128; the cross-attention variant retains adapters and a similar parameter budget.Source: Table 3 · p. 8, §4.3 · p. 8
  • Entropy, balance, and consistency losses each improve performance; together they yield +7.02% over the LM-only baseline.The losses respectively stabilize routing, promote even expert use, and enhance inter-expert alignment.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 ANALYSIS

  • Dropping one expert at inference reduces average ID accuracy by only −0.57 compared with full MoT, as the remaining routed peers can still be integrated.The router and interaction layers remain unchanged from seven-expert training.Source: §4.4 · pp. 8–9, Table 5 · p. 9
  • Accuracy rises as the enabled expert pool grows from one to seven, and Top-K=3 remains effective beyond three experts without retraining.Figure 6c shows scaling after training once on the full pool.Source: §4.4 · p. 9, Figure 6 · p. 9

5 CONCLUSION

MoT combines global routing with latent-level collaboration among heterogeneous LLMs, using frozen expert backbones and single-pass inference. The paper evaluates a seven-model pool and describes its default configuration and training setup.

5 CONCLUSION

  • MoT combines global routing and latent-level collaboration through lightweight interaction layers, keeping expert backbones frozen and enabling single-pass inference.The conclusion reports state-of-the-art performance for multi-LLM systems on diverse ID and OOD benchmarks.Source: §5 · p. 9

A IMPLEMENTATION DETAILS

  • The default MoT uses Q=8 stacks, top-K=3 experts, a 2048-dimensional shared latent space, and eight attention heads; training runs for 50 epochs.The router uses a frozen DeBERTaV3-base encoder; training uses AdamW and A100 80 GB GPUs, with inference on RTX A6000 48 GB GPUs.Source: A IMPLEMENTATION DETAILS · p. 14

A.1 MODELS

  • The seven 7B–8B decoder-only experts span Mistral and Llama-3, mixing general, aligned, domain-specialized, and multilingual variants.The pool includes five Mistral-family models and two Llama-3-family models.Source: §A.1 · p. 14

A.2 DATASETS

The evaluation trains on five in-distribution benchmarks and tests generalization on three datasets unseen during training. Baselines span heuristic, clustering-based, and supervised routing approaches.

A.2 DATASETS

  • Training uses five ID benchmarks—MMLU, GSM8K, CMMLU, ARC-C, and HumanEval—while PreAlgebra, MBPP, and C-Eval are reserved for OOD evaluation.Evaluation uses held-out ID tests; the OOD datasets are unseen during training.Source: §A.2 · p. 14

A.3 BASELINES

  • Baselines include cosine-similarity classification, majority voting, clustering-based LoraRetriever and Avengers, and supervised ZOOTER and RouterDC.Avengers uses ensemble-style sampling and voting; RouterDC uses dual contrastive learning.Source: §A.3 · p. 14
  • The baselines span training-free heuristics and supervised routers under a consistent evaluation protocol.Methods use the same expert pool and datasets for evaluation and training.Source: §A.3 · p. 14

B TRAINING STRATEGY

Training combines a primary-expert language-modeling objective with router and collaboration regularizers, while keeping expert backbones frozen. At inference, selected experts maintain latent trajectories and the primary expert produces the output through cross-attention.

B TRAINING STRATEGY

  • The primary expert is trained with autoregressive negative log-likelihood, while only the router, projectors, and shared cross-attention weights are updated.Expert backbones remain frozen.Source: B TRAINING STRATEGY · pp. 14–15
  • Router entropy and a squared coefficient-of-variation penalty encourage exploration and more uniform expert utilization.The entropy term discourages collapse onto a few experts; the load-balancing term penalizes variation in activation frequency.Source: B TRAINING STRATEGY · p. 15
  • Straight-through relaxation uses hard Top-K selections in the forward pass and soft router probabilities for backward gradients.Gumbel-perturbed scores provide the SoftTopK relaxation while preserving discrete routing at inference.Source: B TRAINING STRATEGY · p. 15
  • Routing-consistency regularization samples two perturbed selections and minimizes token-wise KL divergence between their output distributions.The objective combines this term with the language-modeling and other routing losses.Source: B TRAINING STRATEGY · p. 15
  • Training uses AdamW with weight decay, gradient clipping, warmup, and fixed loss weights λent = 0.01, λbal = 0.01, and λcon = 0.05.These are the stated defaults used in the experiments.Source: B TRAINING STRATEGY · p. 15

C INFERENCE

  • At inference, the router selects active experts and the highest-scoring becomes primary; its query attends over the active experts’ cached representations.During decode, each active expert advances its own token stream, but only the primary expert’s token is appended to the output.Source: C INFERENCE · pp. 16–17

D RESULTS FOR NUMBER OF INTERACTION LAYERS

The section examines how the number of uniformly spaced interaction-layer stacks affects ID and OOD accuracy while holding the other experiment settings fixed.

D RESULTS FOR NUMBER OF INTERACTION LAYERS

  • Table 6 varies the number of uniformly spaced interaction-layer stacks Q and measures ID and OOD accuracy, keeping the expert pool, router, and decoding settings fixed.The supplied passage identifies the comparison setup but gives no accuracy values.Source: Table 6 · p. 18

E RESULTS FOR INTERACTION LAYER INSERTION DEPTH

The section describes an insertion-depth comparison, sequential MoT-style compositions, and the trainable-parameter overhead of MoT configurations.

E RESULTS FOR INTERACTION LAYER INSERTION DEPTH

  • Table 7 compares shallow, intermediate, deep, and uniformly spaced insertion of Q=4 interaction layers using accuracy.The table caption specifies earliest, middle, and latest halves for the three depth placements, with uniform spacing as the control.Source: Table 7 · p. 18

F RESULTS FOR NUMBER OF EXPERTS

  • Table 8 reports benchmark accuracy for sequential MoT-style compositions and gives an average across tasks.The supplied caption does not state the configurations’ results.Source: Table 8 · p. 18

G PARAMETER OVERHEAD

  • Added parameters scale linearly with expert count through projectors and with stack count, and quadratically with shared latent dimension through cross-attention.The projector terms are weighted by expert hidden sizes.Source: G PARAMETER OVERHEAD · p. 19
  • MoT adds 0.8% to 3.4% trainable parameters relative to the 45.75B frozen expert pool; the base configuration uses Q=8 and Top-K=3.The small configuration uses Q=6, ds=1024, hr=128, and Top-K=3.Source: Table 10 · p. 19, G PARAMETER OVERHEAD · p. 19

H TRAINING DETAILS

The training details state that expert backbones are frozen and identify the trainable components and model pool. Separate tables document the base configuration, optimization, loss weights, and decoding settings.

H TRAINING DETAILS

  • All expert backbones are frozen; training updates only the router, per-expert forward/reverse projectors, and shared cross-attention weights.The listed pool includes Mistral-7B-v0.1, MetaMath-Mistral-7B, zephyr-7b-beta, dolphin-2.6-mistral-7b, and Chinese-Mistral-7B-v0.1.Source: H TRAINING DETAILS · p. 20
  • The cited passages list meta-llama/Meta-Llama-3-8B and cognitivecomputations/dolphin-2.9-llama3-8b as entries.Source: H TRAINING DETAILS · p. 20
  • Table 11 gives the MoT base configuration used in training and says it matches the main experiments.Tables 12–14 cover optimization hyperparameters, loss weights, and decoding/evaluation settings, respectively.Source: Table 11 · p. 20, Table 12 · p. 20, Table 13 · p. 20

I ADDITIONAL EXPERIMENTS

This appendix evaluates MoT on benchmarks excluded from its training data and reports gains over AVENGERS; it also records the experiments’ practical scale boundary.

I ADDITIONAL EXPERIMENTS

  • Training allocates 35% of updates to commonsense and multi-choice QA, 25% each to math and code, and 15% to reading comprehension.Domains are sampled with probabilities (0.35, 0.25, 0.25, 0.15), followed by temperature-based dataset sampling at temperature 2.0.Source: I ADDITIONAL EXPERIMENTS · p. 21
  • Evaluation reports accuracy on benchmarks not used for MoT training, using the same seven 7B to 8B experts as training.The additional experiments separate training and evaluation datasets to address possible overlap.Source: I ADDITIONAL EXPERIMENTS · p. 21
  • MoT (Base) leads AVENGERS by +0.42 on the baseline-defined ID average (55.97 vs. 55.55) and +1.95 on OOD (47.35 vs. 45.40), with all evaluation datasets unseen in MoT training.Per-benchmark gains include MMLU (+0.55), CMMLU (+0.10), HumanEval (+0.60), PreAlgebra (+1.10), and C-EVAL (+2.80).Source: Table 15 · p. 21, I ADDITIONAL EXPERIMENTS · p. 21

J USE OF LARGE LANGUAGE MODELS

  • The authors used a language model for grammar, phrasing, LaTeX formatting, and routine table editing, reviewing and editing all suggestions.No proprietary or sensitive data were provided to the model.Source: J USE OF LARGE LANGUAGE MODELS · p. 21

K LIMITATIONS

  • Experiments use a modest pool of frozen experts with limited parameter counts and interaction depth to keep compute, memory, and latency practical.Scaling the expert pool and sizes, varying interaction placement and training, and mapping quality–cost trade-offs are left for future work.Source: K LIMITATIONS · p. 22
Loading 2509.21164v1…