Source-linked AI summary
Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers
Shalev Lifshitz, Sheila A. McIlraith, Yilun Du
TL;DR
Scaling test-time compute beyond candidate sampling remains important as scaling model size and high-quality training data becomes more costly. This paper introduces Multi-Agent Verification with Aspect Verifiers and BoN-MAV, finding that increasing diverse verifiers improves performance across domains and generators, supports weak-to-strong generalization, and enables self-improvement.
Problem
Scaling model size and obtaining unseen high-quality pretraining data are increasingly costly, motivating methods to improve LLM performance by scaling computation at test time.
Method
Multi-Agent Verification combines multiple verifiers, using off-the-shelf Aspect Verifiers for binary approvals and BoN-MAV to select among sampled outputs by approval counts.
Results
Increasing diverse verifiers improves test-time performance across multiple domains and generator LLMs, enables weak-to-strong generalization, and facilitates self-improvement.
Takeaways & Limitations
Scaling the number of verifiers is a promising new dimension for test-time compute, while BoN-MAV is one simple approach within multi-agent verification.
Takeaways & Limitations
The investigation is limited to 20 Aspect Verifiers based on two base LLMs, with verifier design constrained by the ability to devise diverse strategies and relevant aspects.
Abstract
from arXiv · showhide
By utilizing more computational resources at test-time, large language models (LLMs) can improve without additional training. One common strategy uses verifiers to evaluate candidate outputs. In this work, we propose a novel scaling dimension for test-time compute: scaling the number of verifiers. We introduce Multi-Agent Verification (MAV) as a test-time compute paradigm that combines multiple verifiers to improve performance. We propose using Aspect Verifiers (AVs), off-the-shelf LLMs prompted to verify different aspects of outputs, as one possible choice for the verifiers in a MAV system. AVs are a convenient building block for MAV since they can be easily combined without additional training. Moreover, we introduce BoN-MAV, a simple multi-agent verification algorithm that combines best-of-n sampling with multiple verifiers. BoN-MAV demonstrates stronger scaling patterns than self-consistency and reward model verification, and we demonstrate both weak-to-strong generalization, where combining weak verifiers improves even stronger LLMs, and self-improvement, where the same base model is used to both generate and verify outputs. Our results establish scaling the number of verifiers as a promising new dimension for improving language model performance at test-time.
1 Introduction
The paper proposes scaling test-time compute by increasing the number of verifiers, addressing limitations of reward-model verification with Aspect Verifiers and Multi-Agent Verification. BoN-MAV combines best-of-n sampling with multiple aspect verifiers and supports improved scaling, weak-to-strong generalization, and self-improvement.
- Motivation: Scaling test-time compute increases inference computation without simply scaling model parameters or training data.The paper frames this as an alternative response to the rising cost of model scaling and high-quality pretraining data.
- Motivation: Reward-model verifiers require expensive preference-data training and produce uncalibrated scores that are difficult to combine across heterogeneous models.These limitations restrict straightforward scaling in the number and type of verifiers.
- Aspect Verifiers: Aspect Verifiers are off-the-shelf LLMs prompted to issue binary approvals for specific output aspects, enabling simple voting without additional training.Verifier diversity can vary by base LLM, aspect, and verification strategy.
- BoN-MAV: BoN-MAV samples candidate outputs, gathers binary approvals from multiple aspect verifiers, and selects the output receiving the most approvals.It scales test-time compute along both the number of candidates and the number of verifiers.
- Results: Across multiple domains and LLMs, BoN-MAV shows stronger scaling patterns than reward-model verification and self-consistency while enabling weak-to-strong generalization and self-improvement.The same base LLM can serve as both generator and verifier, and many small verifiers can improve stronger generators.
2 Multi-Agent Verification
Multi-Agent Verification (MAV) combines multiple verifiers to evaluate generator outputs, using Aspect Verifiers as train-free, composable building blocks. BoN-MAV aggregates binary approvals to select among sampled candidates while scaling test-time compute through both candidate count and verifier count.
- MAV combines multiple verifiers to evaluate outputs from a generator LLM, creating a new test-time scaling dimension.
- Aspect Verifiers: Aspect Verifiers are off-the-shelf LLMs prompted to assess specific output aspects with binary True/False approvals.They can vary by base LLM, verified aspect, and verification strategy.
- Aspect Verifiers: Aspect Verifiers address reward-model limitations by requiring no additional training and enabling simple voting across heterogeneous models and training data.Their binary approvals are directly combinable even when verifiers use different models or datasets.
- BoN-MAV: BoN-MAV samples n candidate outputs, collects binary approvals from m aspect verifiers, and selects the candidate with the most approvals.This combines best-of-n sampling with multi-agent verification using equal-weight voting.
- BoN-MAV: BoN-MAV scales test-time computation by increasing either the number of sampled outputs n or the number of verifiers m = |M|.Both generation and verification can be parallelized.
- Results: Across eight generator LLMs and four domains, BoN-MAV outperforms self-consistency in nearly all cases and generally outperforms reward-model verification.It performs better than reward-model verification on MATH and MMLU-Pro, with comparable results on GPQA (diamond) and HumanEval.
3 Experiments
The experiments evaluate BoN-MAV across four domains and multiple generator LLMs, scaling both sampled candidates and verifier count. Results show stronger scaling than key baselines, gains from additional verifiers, weak-to-strong generalization, and self-improvement.
- Experimental Setup: BoN-MAV is evaluated across four domains and eight generator LLMs against reward model verification and self-consistency.Each model is evaluated with n = 16 candidate outputs per question.
- Scaling Candidate Outputs: BoN-MAV scales more effectively than self-consistency across all domains and exceeds reward model verification on MATH and MMLU-Pro.Its scaling is comparable to reward model verification on GPQA and HumanEval.
- Scaling Verifier Count: Increasing verifier count improves accuracy in most cases, with gains up to 10% for large LLMs and up to 20% for small ones.Improvement varies by domain and model, sometimes initially decreasing before rising with additional verifiers.
- Generalization and Self-Improvement: Weaker aspect verifiers improve stronger generators, and self-verification improves models when the same base LLM generates candidates and performs verification.The weak-to-strong setting uses Gemini-1.5-Flash and GPT-4o-mini verifiers for stronger generators, while self-improvement reuses the generator model.
- Verifier Analysis: Using diverse verifier sets generally outperforms repeatedly querying the single best-performing verifier.Verifier subsets are engineered for domain-specific validation performance, while combining all available verifiers remains competitive with both baselines.
4 Discussion
MAV scales test-time compute by increasing the number of verifiers, but the paper identifies constraints in verifier diversity, aggregation, generator choice, and training.
- MAV scales test-time compute through multiple verifiers, enabling more effective evaluation, weak-to-strong generalization, and self-improvement.
- The study evaluates only 20 aspect verifiers built from two base LLMs, with diversity constrained by manually designed strategies and aspects.
- Simple majority voting ignores verifier confidence, relevance, and other verifiers’ decisions or feedback.
- BoN-MAV uses one generator LLM, leaving multi-generator sampling as an unexplored direction.
- The work does not investigate fine-tuning the generator on outputs selected by MAV.
- MAV may support safety and oversight by assigning different verifiers to harmful-content, alignment, and reasoning-transparency properties.
5 Related Works
Related work situates MAV within test-time compute methods using additional generation, verifier-guided selection, search, and multi-agent interaction to improve reasoning.
- Scaling Test-Time Compute: Test-time compute research increases inference resources through methods such as scratchpads, chain-of-thought prompting, self-consistency, and majority voting.
- Scaling Test-Time Compute: Best-of-n sampling generates n candidates and selects the highest-scoring output using a verifier.
- Scaling Test-Time Compute: Verifier-guided methods include chain-of-thought scoring, prompted value functions, environment feedback, and trained value functions.
- Multi-Agent Reasoning with Language Models: Multi-agent reasoning research studies language-model debate and multi-agent discourse to enhance reasoning and scalable oversight.
6 Conclusion
The paper introduces MAV and BoN-MAV as a framework for scaling test-time compute along both candidate-generation and verifier-count dimensions. Results indicate that diverse verifier ensembles improve performance across settings and support weak-to-strong generalization and self-improvement.
- MAV combines multiple verifiers with test-time generation, adding verifier count as an orthogonal scaling dimension to candidate sampling.
- Aspect Verifiers are off-the-shelf LLMs requiring no additional training and can combine signals from models differing in architecture, data, training, or prompts.
- BoN-MAV combines best-of-n sampling with multiple aspect verifiers as a simple MAV algorithm.
- Increasing diverse verifiers improves test-time performance across multiple domains and generator LLMs, enables weak-to-strong generalization, and facilitates self-improvement.
- The authors expect better-engineered verifiers and more nuanced aggregation strategies to produce stronger scaling patterns.
A.1 Aspect Verifier Subsets
The appendix lists 20 aspect verifiers and records which are selected for each domain-specific verifier subset.
- Table 5 lists all 20 aspect verifiers in M and identifies the verifiers selected for each domain-specific subset Md.
- The table contains 20 combinations of base models, verification aspects, and verification strategies, with 10 combinations per base model.
- Checkmarks indicate subset membership, and the bottom row reports the number of verifiers |Md| for each domain.
A.2 Generator LLMs
The evaluation uses eight generator LLMs spanning closed-source and open-source models. The model set includes four models in each category, restricted to releases before September 2024.
- The evaluation includes eight generator LLMs.
- Four generators are closed-source models, including Gemini, GPT-4o mini, and GPT-4o variants.
- Four generators are open-source models, including Mistral-7B, Llama-3.1-8B, and Gemma-2 models.
- The generator models were restricted to those released before September 2024.
A.3 Reward Model Baseline
The reward-model baseline uses a pretrained open-source reward model selected for its RewardBench performance. The setup also specifies consistent generator prompts and modular aspect-verifier prompts across domains and base models.
- BoN-RM uses Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 as its reward-model verifier.The model was the top-scoring open-source 8B reward model on RewardBench at the time of writing.
- The pretrained reward model outperformed numerous larger models, including 70B and 340B models, while running on academic-scale compute.
- Generator LLMs use one consistent prompt format, with content varied by domain.
- Aspect-verifier prompts combine a domain-dependent system prompt with a domain-independent aspect-and-strategy prompt.
- This two-part structure supports combining verification methods across domains while maintaining consistent evaluation criteria across base models.
B Additional Results
Using all 20 aspect verifiers without domain-specific tuning remains competitive with reward-model verification and self-consistency. The comparison is reported in Table 6 for BoN-MAV, MAV-all, RM, Cons, and base pass@1 accuracy.
- BoN-MAV with all 20 aspect verifiers remains competitive with reward-model verification and self-consistency.The comparison uses all verifiers without domain-specific engineering or tuning.
- Table 6 labels the untuned all-verifier configuration as MAV-all.
- Table 6 compares MAV-all, reward model verification, self-consistency, and the generator's base pass@1 accuracy.
C Additional Illustrations
Additional illustrations show aspect verifiers evaluating MATH, HumanEval, and GPQA solutions with varied aspects, strategies, and base models. The prompt tables define domain-specific contexts and aspect-strategy combinations for these evaluations.
- Additional Illustrations: Additional examples cover MATH, HumanEval, and GPQA solutions evaluated by multiple aspect verifiers.
- Additional Illustrations: On an incorrect HumanEval solution, two verifiers identify the error while one incorrectly approves it.
- Additional Illustrations: On a correct GPQA solution, verifier behavior varies with base model, verified aspect, and verification strategy.
- Prompt Specifications: Generator prompts use one consistent prompt for each domain across generator LLMs.
- Prompt Specifications: The verifier prompts are organized by aspect-strategy combination and follow domain-specific system prompts.