Source-linked AI summary
Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, Jingren Zhou
TL;DR
LLM ensembles could exploit heterogeneous model expertise, but reward-model ranking methods impose substantial computation overhead. ZOOTER trains a reward-guided router from query-level reward supervision, with tag-based enhancement for reward uncertainty. Across broad benchmarks, it outperforms the best single model on average and reward-model ranking ensembles with lower overhead.
Problem
LLM ensemble methods can exploit heterogeneous expertise, but existing reward-model ranking approaches require substantial computation.
Method
ZOOTER distills normalized rewards from training queries to train a routing function, using tag-based label enhancement to reduce uncertainty noise.
Results
ZOOTER outperforms the best single model on average and reward-model ranking ensembles with significantly fewer computation overhead across 26 benchmark subsets.
Takeaways & Limitations
Reward-model supervision can reveal latent expertise and support efficient query routing among off-the-shelf LLMs.
Takeaways & Limitations
Reward-model scalar uncertainty can introduce noise into routing training, and GPT-4 judgments may disagree with human judgments.
Abstract
from arXiv · showhide
The complementary potential of Large Language Models (LLM) assumes off-the-shelf LLMs have heterogeneous expertise in a wide range of domains and tasks so that an ensemble of LLMs can achieve consistently better performance. Existing ensemble methods for LLMs mainly focus on reward model ranking of outputs, leading to significant computation overhead. To combat this issue, we revisit the complementary potential of LLMs and further elaborate it by mining latent expertise with off-the-shelf reward models. We propose Zooter, a reward-guided routing method distilling rewards on training queries to train a routing function, which can precisely distribute each query to the LLM with expertise about it. We also integrate a tag-based label enhancement to mitigate noise from uncertainty when using rewards as silver supervision. Zooter shows computation efficiency in inference as it introduces only a minor computation overhead of a routing function compared with reward model ranking methods. We evaluate Zooter on a comprehensive benchmark collection with 26 subsets on different domains and tasks. Zooter outperforms the best single model on average and ranks first on 44% of tasks, even surpassing multiple reward model ranking methods.
1 Introduction
LLMs may possess complementary expertise across domains and tasks, but existing ensemble approaches face substantial inference costs. ZOOTER addresses this by using reward-guided routing and enhanced silver labels to assign queries to capable models efficiently.
- Off-the-shelf LLMs can have heterogeneous strengths, making ensembles potentially better than a single best-on-average model across diverse tasks.
- Existing LLM ensemble methods require substantial inference computation, limiting scalability in low-resource settings.
- ZOOTER distills reward-model signals from training queries to train a router that assigns each query to a model with relevant latent expertise.
- Tag-based label enhancement reduces noise caused by uncertainty when reward-model outputs serve as silver supervision.
- ZOOTER is evaluated across four benchmark groups containing 26 subsets and outperforms reward-model ranking methods with significantly lower computation overhead.
2 Related Works
Related work studies LLM ensembling, routing, and reward-guided generation, but many approaches incur overhead by processing candidate models or benchmark supervision. ZOOTER instead distills expertise from rewards using query-only training data for efficient routing.
- LLM ensemble methods aim to combine off-the-shelf models for consistently better performance across diverse downstream tasks.
- Prior approaches include output fusion, sequential inference, supervised expert fusion, and learned routing strategies.
- Many concurrent methods require candidate generations or forward passes to obtain representations, creating significant computation overhead.
- ZOOTER distills rewards on predefined queries and can be developed without golden responses, emphasizing efficiency in data and computation.
- Reward models are also used to improve generation by ranking alternatives, including reasoning paths.
3 Methods
ZOOTER routes queries to expert LLMs by distilling reward-model judgments from diverse training queries, while tag-based enhancement reduces reward uncertainty. This replaces inference-time ranking of all candidate outputs with a routing function that selects one expert LLM and adds only minor routing overhead.
- Query Routing: Query routing assigns each query to an LLM predicted to have expertise in it, avoiding output generation from all candidates during inference.The routing function predicts query–expert associations directly from the query.
- Inference Efficiency: During inference, ZOOTER uses the routing function to select one expert LLM, adding only a small computation overhead compared with ranking all candidate outputs.The training queries require no response annotations, although they are expected to be diverse for generalization.
- Reward Distillation: Normalized reward distributions serve as silver supervision for training the routing function with a Kullback-Leibler divergence loss.The loss is L(q_i, r_i) = KL(Z(q_i), softmax(r_i)).
- Reward Distillation: ZOOTER estimates model expertise by running all candidate LLMs on diverse training queries and scoring their responses with an off-the-shelf reward model.The resulting scalar rewards provide supervision for learning the router.
- Tag-based Label Enhancement: Instruction tags aggregate rewards across similarly tagged queries, then combine tag-wise and sample-level rewards to smooth and denoise supervision.The combination uses β for the trade-off between coarse-grained tag-wise rewards and fine-grained sample-level rewards.
4 Experiments
ZOOTER is evaluated against single-model and reward-model-ranking baselines across four benchmark groups and 26 subsets. It improves consistency while adding only a small routing overhead, though reward-model ranking remains stronger in some settings.
- Experimental Setup: The evaluation covers four benchmark groups, including AlpacaEval, FLASK, MT-Bench, MMLU, GSM8K, and HumanEval, across diverse tasks and domains.MT-Bench uses first-turn queries for routing but evaluates multi-turn conversations; MMLU, GSM8K, and HumanEval supplement GPT-4-judged benchmarks.
- Experimental Setup: Mean task rank and uplift rate summarize cross-benchmark consistency, with lower MTR and higher uplift indicating stronger performance.The metrics are reported over 26 evaluation subsets.
- Main Results: RMR with UltraRM achieves the strongest aggregate ranking, with MTR 1.53 and the best model across 72% of subtasks.QwenRM ranks second and performs similarly with smaller parameter size; several RMR methods outperform the best model on average.
- Main Results: ZOOTER outperforms the best model on average on AlpacaEval, MT-Bench, and the reported benchmark aggregate, matches it on FLASK, and improves MT-Bench by 0.39.It ranks first on 44% of subtasks, compared with 31% for the best model on average.
- Main Results: ZOOTER uses only an 86M ranker for routing overhead and outperforms several RMR baselines with substantially lower computation, but remains behind RMR with QwenRM overall.Its comparison with QwenRM is stronger on AlpacaEval than in general.
- Analysis: Reward uncertainty motivates tag-based label enhancement: lower reward entropy tends to accompany higher MT-Bench scores, and β = 0.3 gives the best reported performance.The ablation supports combining sample-level and tag-level rewards rather than using sample-level rewards alone.
5 Conclusion
The paper proposes ZOOTER, an efficient reward-guided routing method for assembling off-the-shelf LLMs. Comprehensive evaluation shows it can outperform the best single model and reward-model-ranking ensembles with significantly lower computation overhead.
- ZOOTER is an efficient reward-guided routing method for ensemble off-the-shelf LLMs.
- ZOOTER outperforms the best single model on average while using significantly less computation overhead than reward-model-ranking ensembles.
- The paper provides evidence for the complementary potential of open-source LLMs and the effectiveness of LLM ensembles.
- Future work includes interpreting the latent expertise of each LLM.
A Datasets
DIVINSTRUCT is a diverse instruction dataset assembled from multiple open-source datasets and carefully decontaminated against the evaluated benchmarks.
- DIVINSTRUCT combines instruction data from multiple open-source datasets.
- The dataset applies careful decontamination across all benchmarks evaluated in this work.