The Majority is not always right: RL training for solution aggregation

1 Introduction

• Majority-based aggregation can miss correct minority answers and overlook partial correctness that could be combined across flawed reasoning traces.Selecting the most frequent output or maximizing model score therefore may not yield the optimal result.Source: §1 · p. 1
• AggLM samples multiple solutions, then trains a reasoning model with RLVR to reconcile, correct, and combine their reasoning into a final answer.This treats aggregation as a learned reasoning skill rather than a fixed heuristic.Source: §1 · p. 1
• On AIME25, AggLM-1.7B reaches 50% versus 35% for Qwen3-1.7B and 45% for majority voting, while beating rule-based and 72B reward-model baselines across four datasets.It also remains the top performer when aggregating solutions from stronger Qwen3-8B models or from non-thinking mode, and uses fewer tokens than majority voting with more samples.Source: §1 · pp. 1–2
• Balancing easy and hard training examples is critical, and AggLM's gains over majority voting are especially substantial when candidate sets are small and diverse.The method is also substantially more token-efficient than majority voting with larger numbers of solutions.Source: §1 · p. 2

2 Related Work

• Rule-based voting can amplify errors when correct solutions occur only in minority modes, despite often yielding reliable gains.Variants include dynamic sample counts and heuristic filters.Source: §2 · p. 2
• Reward-model selection re-ranks candidates but can introduce more regression errors than improvements, while Universal Self-Consistency prompts a model to choose the most coherent sample.Model-based methods broadly use a separate scorer or prompt the LLM to compare and consolidate candidates.Source: §2 · p. 2
• Unlike prompting alone, AggLM trains a reasoning-focused aggregator with RL to synthesize answers; concurrent SSA also uses RL, while the authors point to reasoning-oriented models and balanced training mixtures as important ingredients.The paper's ablations support the importance of balancing the training mixture in its setting.Source: §2 · pp. 2–3

3 AggLM

• The aggregator learns to select a correct candidate already present or synthesize a new correct solution by detecting mistakes, filling gaps, and combining complementary partial ideas.Training uses problems with verifiable solutions and reinforcement learning from verifiable rewards.Source: §3 · p. 3
• The formulation separates a solution model that independently samples m candidates from an aggregation model that reads the problem and candidates and outputs a final solution.The solution and aggregation models may share parameters or be different models; main experiments keep the off-the-shelf solution generator fixed.Source: §3 · p. 3
• Training data groups sampled solutions into sets, with increasing s adding diversity to the answer combinations seen for each problem.Examples are hard when the majority answer is wrong and easy otherwise; the training mixture keeps all hard examples and adds a proportion of easy ones to avoid undertraining minority-answer recovery while retaining non-sparse rewards.Source: §3 · p. 3
• The aggregation policy is optimized with GRPO using a binary reward.The supplied method passage identifies the update as the standard GRPO update.Source: §3 · p. 3

4.1 Training

• AggLM-1.7B is initialized from Qwen3-1.7B and trained for one epoch on 446,220 examples, using a p=50% data mixture built from around 40,000 DeepScaler math problems.The training data is constructed from 128 independent Qwen3-1.7B thinking-mode solutions per problem, divided into 16 sets of eight.Source: §4.1 · pp. 3–4
• The aggregation prompt template instructs the model to produce reasoning and a final aggregated solution, both of which are trained by RL.The template is used to prompt the LLM to aggregate candidate solutions.Source: Figure 2 · p. 4

4.2 Evaluation

• Evaluation uses Qwen3-1.7B in thinking and non-thinking modes and Qwen3-8B in thinking mode across AIME24, AIME25, HMMT24, and HMMT25.Each benchmark dataset contains 30 examples.Source: §4.2 · p. 4
• For each problem, 128 sampled solutions are divided into 16 sets of eight; the aggregator generates four answers per set, and scores average set-level pass@1 across problems.The protocol is designed to obtain reliable signals from relatively small evaluation datasets.Source: §4.2 · p. 4
• Baselines include majority voting, best-of-N, reward-model selection and weighted voting, prompted aggregation without fine-tuning, and the original model's pass@1 and pass@8.The tables report results for eight sampled solutions from Qwen3-1.7B thinking mode and Qwen3-8B thinking mode, respectively.Source: §4.2 · p. 4, Table 1 · p. 5, Table 2 · p. 5

Evaluation on in-distribution solutions

• AggLM-1.7B is best on all four in-distribution benchmarks, with consistent three-to-seven-point gains over majority voting and prompted Qwen3-1.7B.These are 1.7B-thinking solutions, the same solution model used to produce the aggregator’s training data.Source: §5 · p. 5

Evaluation on solutions from stronger models

• Despite training only on 1.7B solution distributions, AggLM-1.7B remains the top performer across datasets with solutions from a stronger 8B thinking model.It also outperforms majority voting, reward-model selection, and prompted generative aggregation without RL.Source: §5 · p. 5

Evaluation on non-thinking solutions

• On eight non-thinking Qwen3-1.7B solutions, AggLM remains top across datasets, while reward-model selection improves over majority voting in this lower-signal regime.AggLM also outperforms majority voting, reward-model selection, and prompted generative aggregation without RL.Source: §5 · pp. 5–6, Table 3 · p. 6

Scaling via the number of solutions

• On AIME25, HMMT24, and HMMT25, AggLM-1.7B with eight solutions outperforms majority voting with sixteen, and its performance rises more steeply as candidate count increases.Although trained at k=8, AggLM generalizes to smaller and larger k.Source: §6 · p. 6, Figure 3 · p. 7

Effect of majority answer size

• AggLM-1.7B gains most when the majority answer is infrequent, while matching majority voting when that answer is frequent.The majority answer size is the count of occurrences of the most frequent answer.Source: §6 · p. 6, Figure 4 · p. 8

Ablating data mixtures

• A moderate share of easy sets (5–50%) consistently improves accuracy; hard-only training is suboptimal, while retaining all available data brings only marginal improvement over an untrained aggregator.Performance is stable across the 5–50% range when hard examples are retained.Source: §6 · p. 7

Ablating number of solution sets s

• Increasing the number of solution sets per problem yields only slight performance gains, making fewer sets a reasonable way to reduce the training budget.The ablation varies the number of sets s while keeping the number of candidates per set fixed.Source: §6 · p. 7, Table 5 · p. 7

Aggregation versus extra data and shared models

• Fine-tuning the solution model on the aggregator's training data produces small or negative improvements and remains far below the trained aggregator across benchmarks; multitasking gives performance close to a separate aggregator.The comparison tests both whether extra data explains the gains and whether one model can perform solution generation and aggregation.Source: §6 · p. 8, Table 6 · p. 9

Token usage

• The aggregator generates roughly one-third as many tokens per inference as solution models, while increasing majority-vote k beyond AggLM's setting would require substantially higher token usage for a comparable boost.The token comparison is reported in Table 7.Source: §6 · p. 8, Table 7 · p. 9

• The proposed method learns to aggregate solutions using RLVR as a test-time scaling approach for hard reasoning problems.Source: §7 · p. 9
• Experiments report improved performance over strong baselines, including self-consistency decoding and weighted majority voting with state-of-the-art math reward models.Source: §7 · p. 9
• The approach generalizes to input distributions from different, including stronger, solution models and to numbers of solutions unlike those used in training.Source: §7 · p. 9

A Qualitative Analysis

• In Example B, AggLM-1.7B verifies and compares the supplied solutions, then chooses the correct one.The example includes one correct solution in the prompt.Source: A Qualitative Analysis · pp. 11–12
• In Example A, AggLM-1.7B synthesizes a correct solution even though none of the prompt solutions is correct.The example gives a=507 and b=3, with a+b=510.Source: A Qualitative Analysis · p. 11
• For the coin-change problem, the worked solution counts 390 failing values and derives 610 values of N for which the greedy algorithm succeeds.The count subtracts the failing values from the 1,000 values in the range.Source: A Qualitative Analysis · pp. 12–13