Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers

Motivation

• Scaling test-time compute increases inference computation without simply scaling model parameters or training data.The paper frames this as an alternative response to the rising cost of model scaling and high-quality pretraining data.Source: §1 · p. 1
• Reward-model verifiers require expensive preference-data training and produce uncalibrated scores that are difficult to combine across heterogeneous models.These limitations restrict straightforward scaling in the number and type of verifiers.Source: §1 · pp. 1–2

Aspect Verifiers

• Aspect Verifiers are off-the-shelf LLMs prompted to issue binary approvals for specific output aspects, enabling simple voting without additional training.Verifier diversity can vary by base LLM, aspect, and verification strategy.Source: §1 · p. 2

BoN-MAV

• BoN-MAV samples candidate outputs, gathers binary approvals from multiple aspect verifiers, and selects the output receiving the most approvals.It scales test-time compute along both the number of candidates and the number of verifiers.Source: §1 · p. 2

Results

• Across multiple domains and LLMs, BoN-MAV shows stronger scaling patterns than reward-model verification and self-consistency while enabling weak-to-strong generalization and self-improvement.The same base LLM can serve as both generator and verifier, and many small verifiers can improve stronger generators.Source: §1 · p. 2

• MAV combines multiple verifiers to evaluate outputs from a generator LLM, creating a new test-time scaling dimension.Source: §2 · p. 2

Aspect Verifiers

• Aspect Verifiers are off-the-shelf LLMs prompted to assess specific output aspects with binary True/False approvals.They can vary by base LLM, verified aspect, and verification strategy.Source: §2.1 · pp. 3–4
• Aspect Verifiers address reward-model limitations by requiring no additional training and enabling simple voting across heterogeneous models and training data.Their binary approvals are directly combinable even when verifiers use different models or datasets.Source: §2.1 · p. 3

BoN-MAV

• BoN-MAV samples n candidate outputs, collects binary approvals from m aspect verifiers, and selects the candidate with the most approvals.This combines best-of-n sampling with multi-agent verification using equal-weight voting.Source: §2.3 · p. 4, §2.2 · p. 4
• BoN-MAV scales test-time computation by increasing either the number of sampled outputs n or the number of verifiers m = |M|.Both generation and verification can be parallelized.Source: §2.3 · p. 4

Results

• Across eight generator LLMs and four domains, BoN-MAV outperforms self-consistency in nearly all cases and generally outperforms reward-model verification.It performs better than reward-model verification on MATH and MMLU-Pro, with comparable results on GPQA (diamond) and HumanEval.Source: Table 1 · p. 5

Experimental Setup

• BoN-MAV is evaluated across four domains and eight generator LLMs against reward model verification and self-consistency.Each model is evaluated with n = 16 candidate outputs per question.Source: §3.1 · pp. 5–6, §3 · p. 5

Scaling Candidate Outputs

• BoN-MAV scales more effectively than self-consistency across all domains and exceeds reward model verification on MATH and MMLU-Pro.Its scaling is comparable to reward model verification on GPQA and HumanEval.Source: Figure 4 · p. 6, §3.1 · p. 7

Scaling Verifier Count

• Increasing verifier count improves accuracy in most cases, with gains up to 10% for large LLMs and up to 20% for small ones.Improvement varies by domain and model, sometimes initially decreasing before rising with additional verifiers.Source: §3.1 · pp. 7–8

Generalization and Self-Improvement

• Weaker aspect verifiers improve stronger generators, and self-verification improves models when the same base LLM generates candidates and performs verification.The weak-to-strong setting uses Gemini-1.5-Flash and GPT-4o-mini verifiers for stronger generators, while self-improvement reuses the generator model.Source: §3.2 · p. 9

Verifier Analysis

• Using diverse verifier sets generally outperforms repeatedly querying the single best-performing verifier.Verifier subsets are engineered for domain-specific validation performance, while combining all available verifiers remains competitive with both baselines.Source: §3.3 · p. 9

• MAV scales test-time compute through multiple verifiers, enabling more effective evaluation, weak-to-strong generalization, and self-improvement.Source: §4 · p. 9
• The study evaluates only 20 aspect verifiers built from two base LLMs, with diversity constrained by manually designed strategies and aspects.Source: §4 · p. 9
• Simple majority voting ignores verifier confidence, relevance, and other verifiers’ decisions or feedback.Source: §4 · p. 10
• BoN-MAV uses one generator LLM, leaving multi-generator sampling as an unexplored direction.Source: §4 · p. 10
• The work does not investigate fine-tuning the generator on outputs selected by MAV.Source: §4 · p. 10
• MAV may support safety and oversight by assigning different verifiers to harmful-content, alignment, and reasoning-transparency properties.Source: §4 · p. 10

Scaling Test-Time Compute

• Test-time compute research increases inference resources through methods such as scratchpads, chain-of-thought prompting, self-consistency, and majority voting.Source: §5 · p. 10
• Best-of-n sampling generates n candidates and selects the highest-scoring output using a verifier.Source: §5 · p. 10
• Verifier-guided methods include chain-of-thought scoring, prompted value functions, environment feedback, and trained value functions.Source: §5 · p. 11

Multi-Agent Reasoning with Language Models

• Multi-agent reasoning research studies language-model debate and multi-agent discourse to enhance reasoning and scalable oversight.Source: §5 · p. 11

• MAV combines multiple verifiers with test-time generation, adding verifier count as an orthogonal scaling dimension to candidate sampling.Source: §6 · p. 11
• Aspect Verifiers are off-the-shelf LLMs requiring no additional training and can combine signals from models differing in architecture, data, training, or prompts.Source: §6 · p. 11
• BoN-MAV combines best-of-n sampling with multiple aspect verifiers as a simple MAV algorithm.Source: §6 · p. 11
• Increasing diverse verifiers improves test-time performance across multiple domains and generator LLMs, enables weak-to-strong generalization, and facilitates self-improvement.Source: §6 · p. 11
• The authors expect better-engineered verifiers and more nuanced aggregation strategies to produce stronger scaling patterns.Source: §6 · p. 11

• Table 5 lists all 20 aspect verifiers in M and identifies the verifiers selected for each domain-specific subset Md.Source: §A.1 · p. 16
• The table contains 20 combinations of base models, verification aspects, and verification strategies, with 10 combinations per base model.Source: Table 5 · p. 16
• Checkmarks indicate subset membership, and the bottom row reports the number of verifiers |Md| for each domain.Source: Table 5 · p. 16

• The evaluation includes eight generator LLMs.Source: §A.2 · p. 16
• Four generators are closed-source models, including Gemini, GPT-4o mini, and GPT-4o variants.Source: §A.2 · p. 16
• Four generators are open-source models, including Mistral-7B, Llama-3.1-8B, and Gemma-2 models.Source: §A.2 · p. 16
• The generator models were restricted to those released before September 2024.Source: §A.2 · p. 16

• BoN-RM uses Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 as its reward-model verifier.The model was the top-scoring open-source 8B reward model on RewardBench at the time of writing.Source: §A.3 · p. 16
• The pretrained reward model outperformed numerous larger models, including 70B and 340B models, while running on academic-scale compute.Source: §A.3 · p. 16
• Generator LLMs use one consistent prompt format, with content varied by domain.Source: §A.4 · p. 17
• Aspect-verifier prompts combine a domain-dependent system prompt with a domain-independent aspect-and-strategy prompt.Source: §A.4 · p. 17
• This two-part structure supports combining verification methods across domains while maintaining consistent evaluation criteria across base models.Source: §A.4 · p. 17

• BoN-MAV with all 20 aspect verifiers remains competitive with reward-model verification and self-consistency.The comparison uses all verifiers without domain-specific engineering or tuning.Source: B Additional Results · p. 17
• Table 6 labels the untuned all-verifier configuration as MAV-all.Source: Table 6 · p. 17
• Table 6 compares MAV-all, reward model verification, self-consistency, and the generator's base pass@1 accuracy.Source: Table 6 · p. 17

Additional Illustrations

• Additional examples cover MATH, HumanEval, and GPQA solutions evaluated by multiple aspect verifiers.Source: C Additional Illustrations · p. 17, Figure 7 · p. 18, Figure 8 · p. 19
• On an incorrect HumanEval solution, two verifiers identify the error while one incorrectly approves it.Source: Figure 8 · p. 19
• On a correct GPQA solution, verifier behavior varies with base model, verified aspect, and verification strategy.Source: Figure 9 · p. 20

Prompt Specifications

• Generator prompts use one consistent prompt for each domain across generator LLMs.Source: Table 7 · p. 21
• The verifier prompts are organized by aspect-strategy combination and follow domain-specific system prompts.Source: Table 8 · p. 21, Table 9 · p. 22, Table 10 · p. 23