PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament

1 Introduction

• PairJudge RM judges two candidate solutions' correctness instead of assigning absolute scores, addressing the arbitrariness and inconsistency of reward-model scores.The approach is combined with a knockout tournament for Best-of-N sampling.Source: §1 · p. 1
• PairJudge RM significantly outperforms baseline reward models on MATH-500 and Olympiad Bench, with 40% to 60% relative improvement on the top 50% challenging MATH-500 problems.The 40% to 60% comparison is against baseline models.Source: §1 · p. 2
• PAIRJUDGE-432K contains 432K pairwise judgments derived from NumiaMath and annotated using gemini-1.5-flash.The dataset is used to train PairJudge RM via supervised fine-tuning.Source: §1 · p. 2

2 Preliminaries

• Best-of-N sampling selects the best of N candidate solutions for a math problem using an external selection mechanism.The preliminaries distinguish outcome and process reward models as the two types of reward models serving this role.Source: §2 · p. 2
• An outcome reward model assigns each candidate a numerical score and selects the candidate with the highest score.It is typically trained on chosen and rejected response pairs to assign the chosen response a higher reward.Source: §2 · p. 2
• A process reward model scores each reasoning step, averages those scores for the solution, and selects the solution with the highest score.It is typically trained to predict whether each reasoning step is correct or incorrect.Source: §2 · p. 3

3.1 Pairwise Judge Reward Model

• PairJudge RM takes a math problem and two candidate solutions and judges whether each solution is correct.It generates chain-of-thought reasoning to verify both solutions before generating their correctness labels.Source: §3.1 · p. 3

3.2 Knockout Tournament

• The tournament groups candidates sharing the same final answer into teams, then pairs candidates from different teams for matches.The tournament continues until one candidate remains or all candidates are from the same team.Source: §3.2 · p. 4
• In each match, a candidate labeled correct advances; if both are labeled correct, one advances at random, and if both are incorrect, both are eliminated.The procedure returns the last remaining candidate as the best solution.Source: §3.2 · pp. 3–4

4 PAIRJUDGE-432K dataset collection

• PAIRJUDGE-432K contains 432K pairwise judgments derived from NumiaMath (LI et al., 2024) and annotated using gemini-1.5-flash.The judgments train PairJudge RM via supervised fine-tuning.Source: §4 · p. 4

4.1 Dataset Format

• Each prompt pairs a math problem with two candidate solutions, and its completion gives chain-of-thought judgments and correctness labels for both.The training data uses a prompt-completion format suited to supervised fine-tuning.Source: §4.1 · p. 4

4.2 Math Problem Collection

• NumiaMath problems with messy formatting, OCR errors, or missing answers were removed, as were multiple-choice, true/false, and proof problems.The stated reasons include avoiding random guessing on selected-answer questions and difficulty verifying proof solutions.Source: §4.2 · p. 4

4.3 Candidate Solution Generation

• For each problem, 24 candidate solutions were generated with Llama-3.1-8B-Instruct using four-shot prompts, temperature 1.0, and top-p 0.5.The decoding settings were chosen to balance diversity and quality.Source: §4.3 · p. 4

4.4 Pairwise Judgment Annotation

• The annotation pipeline retained correct pairwise judgments from 343K tournaments and filtered for instruction compliance, yielding 432K training samples.The tournaments recorded 2.2M comparisons; 1.3M correctly evaluated both solutions before the final filtering.Source: §4.4 · p. 4

5 Experiments

• The experiments assess PairJudge RM with a knockout tournament for Best-of-N sampling at test time.The section introduces the evaluation setup, results, and subsequent analysis.Source: §5 · pp. 4–5

5.1 Experimental Setup

• The evaluation covers MATH-500 and Olympiad Bench with Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, and Qwen2.5-7B-Instruct.Baselines include outcome and process reward models and majority voting.Source: §5.1 · pp. 5–6

5.2 Results

• 6.7% on MATH-500 and 3.9% on Olympiad Bench are PairJudge RM’s average improvements over the strongest baseline model, excluding majority voting.The reported comparison spans datasets and generation models; Table 2 summarizes accuracy for Best-of-16, Best-of-32, and Best-of-64.Source: Table 2 · p. 5, §5.2 · p. 6

5.3 Difficulty Analysis

• 40%–60% relative improvement over baseline models is reported for the hardest 50% of MATH-500 problems.Difficulty is the fraction of incorrect candidate answers, measured with 64 candidates; PairJudge RM outperforms baselines except on the easiest problems.Source: §5.3 · pp. 6–7, Figure 2 · p. 6

6.1 Comparison on Correctness Verification

• PairJudge RM outperforms the Critic Model on correctness verification across both MATH-500 and Olympiad.Table 3 reports accuracy; the supplied text gives no numerical values.Source: §6.1 · p. 7, Table 3 · p. 7

6.2 Comparison on Best-of-N Sampling

• Pairwise judgments directly compare candidates, whereas pointwise Critic Model judgments can leave the better of two solutions marked correct unresolved.The section identifies this difference as a challenge for using the Critic Model in Best-of-N sampling.Source: §6.2 · p. 7
• The section describes majority voting after incorrect candidates are removed and probabilistic scoring that selects the candidate with the highest “correct” token probability.These are presented as two approaches for enabling Critic Model Best-of-N sampling.Source: §6.2 · pp. 7–8
• PairJudge RM consistently outperforms the Critic Model on MATH-500 Best-of-N sampling.The evaluation reuses the models trained in Section 6.1 and uses the MATH-500 test split with Qwen-2.5-7B-Instruct candidate solutions.Source: §6.2 · p. 8
• The Critic Model's probabilistic-score approach underperforms majority voting, and similarly correct solutions can receive scores of 0.0006 and 0.9999.The paper attributes the probabilistic-score result likely to highly polarized scores and notes robustness and stability issues.Source: §6.2 · p. 8, Figure 4 · p. 8

7.1 Test Time Scaling and Best-of-N Sampling

• The knockout method groups candidates with identical answers to avoid unnecessary comparisons and uses generative chain-of-thought judgment rather than discriminative scoring.The passage contrasts this method with tournament-style selection previously introduced for instruction-following tasks.Source: §7.1 · p. 8

7.2 Reward Models and Critic Models

• PairJudge RM judges two responses' correctness simultaneously with chain-of-thought reasoning, unlike pointwise Critic Models and numerically scoring reward models.The related-work discussion describes Best-of-N as generating N candidates and selecting the best via a reward model.Source: §7.2 · p. 8, §7.1 · p. 8

8 Conclusion

• PairJudge RM uses chain-of-thought reasoning to evaluate two candidate solutions simultaneously, while the knockout tournament iteratively eliminates incorrect solutions through pairwise judgments.The approach is presented for Best-of-N sampling and is described as eliminating arbitrary scoring and enabling cross-validation.Source: §8 · p. 8
• The authors create a 432K pairwise-judgment dataset to train PairJudge RM.The conclusion does not specify the dataset construction details in this passage.Source: §8 · p. 9
• Experiments show PairJudge RM significantly outperforms baseline reward models in benchmark Best-of-N sampling.Source: §8 · p. 9

Limitation

• Several rounds of pairwise verification make inference time-consuming and may make the method unsuitable for latency-sensitive applications.The authors note that independent verifications could be parallelized across multiple GPUs to speed inference.Source: Limitation · p. 9

• The experiments use data and models from public sources.Source: Ethical Considerations · p. 9
• No human subjects are involved in the work.Source: Ethical Considerations · p. 9
• Potential pretrained-model hallucination and bias call for caution in real-world applications such as checking students’ homework.Source: Ethical Considerations · p. 9

A Dataset Information

• MATH-500 contains 500 problems, Olympiad Bench contains 8,476 competition problems, and NuminaMath-CoT contains 860k math problems with chain-of-thought solutions.Source: A Dataset Information · p. 11

B Computational Resources

• Training Qwen-2.5-7B-Instruct on PAIRJUDGE-432K used an 8-GPU H100 server for about 24 hours, while dataset construction incurred about $2,000 in Gemini API costs.Source: B Computational Resources · pp. 11–12

Application in Reinforcement Learning

• The paper focuses on test-time Best-of-N sampling but proposes deriving reinforcement-learning reward signals from candidates’ knockout-tournament winning rates as future work.Source: Application in Reinforcement Learning · p. 12

Alternative Tournament Strategies

• Knockout is chosen for its O(N) time complexity, while round-robin, Swiss-system, and double-elimination tournaments are noted as possible alternatives for future exploration.Source: Alternative Tournament Strategies · p. 12

D Potential Improvement

• Potential improvements include a larger judge, more candidate-generation and annotation models to expand training data, and pairing the method with long-chain-of-thought models.Source: D Potential Improvement · p. 12

E Artifacts in Our Research

• The listed research artifacts include PyTorch, Hugging Face Transformers, DeepSpeed, Llama, and Qwen.Source: E Artifacts in Our Research · pp. 12–13

F Experiments Replication; G Prompt Templates; H Math Problem Filtering Criteria

• The authors plan to release code, checkpoints, and data, and report three-run averages; the appendices also provide PairJudge prompts and dataset filtering criteria.Source: F Experiments Replication · p. 13, G Prompt Templates · p. 13, H Math Problem Filtering Criteria · p. 13