Source-linked AI summary
Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3
Natapong Nitarach
TL;DR
Majority voting for mathematical reasoning is limited by correlated errors, motivating a test of whether diverse prompts can reduce that limitation. The paper evaluates Diverse Prompt Mixer across AIMO 3 models and experiments. Prompt-level interventions fail, while model capability dominates and selection-level optimization remains open.
Problem
Correlated errors limit the effective sample size of majority voting, motivating whether different reasoning strategies can reduce error correlation.
Method
The paper tests Diverse Prompt Mixer across three models, 23+ experiments, and 50 IMO-level problems under a fixed competition-compute setting.
Results
Model capability dominates prompt-level inference-time optimization: at equal N=8, the capability gap is 8 points versus prompt effects of ±2 points, and no fixed-model optimization improves over baseline.
Takeaways & Limitations
High-temperature sampling already provides sufficient diversity, so prompt diversity is redundant or harmful; selection-level optimization remains open.
Takeaways & Limitations
The negative result is limited to prompt-level inference-time optimization; selection-level optimization remains a separate open question.
Abstract
from arXiv · showhide
Majority voting over multiple LLM attempts improves mathematical reasoning, but correlated errors limit the effective sample size. A natural fix is to assign different reasoning strategies to different voters. The approach, Diverse Prompt Mixer, is tested on the AIMO 3 competition: 3 models, 23+ experiments, 50 IMO-level problems, one H100 80 GB, 5-hour limit. Every prompt-level intervention fails. High-temperature sampling already decorrelates errors; weaker strategies reduce accuracy more than they reduce correlation. Across an 8-point capability gap at equal N=8 and every optimization tested, model capability dominates. The gap between the best majority-vote score (42/50) and pass@20 (~45.5) is selection loss, not prompt loss. A verifier-based selector could close it. Prompt engineering cannot.
1 Introduction
Majority voting can improve mathematical reasoning when attempts are accurate and independent, but correlated errors reduce its effective sample size. Diverse Prompt Mixer proposes assigning different reasoning strategies to voters, yet the paper concludes the idea does not work.
- Majority voting is the standard approach for competition mathematics and converges toward correctness as attempts grow when p > 0.5 and errors are independent.
- Correlated systematic mistakes mean multiple attempts provide fewer effective votes than their raw count suggests.
- At ρ = 0.3, eight attempts correspond to only 2.6 effective votes.
- Diverse Prompt Mixer assigns different reasoning strategies to voters to reduce error correlation.
- The paper finds that this intuitive decorrelation strategy is wrong.
2 System Architecture
The system is designed for a single-H100, five-hour competition budget, using parallel sampled attempts, tool-integrated reasoning, early stopping, and entropy-weighted voting. Its architecture explicitly allocates time to guarantee completion of all 50 problems.
- The system runs on one H100 80 GB within Kaggle’s five-hour wall-clock limit, without external APIs, multi-GPU setups, or pre-computed solutions.
- The pipeline has five stages: budget allocation, parallel attempts, tool-integrated reasoning, early stopping, and entropy-weighted voting.
- Eight concurrent attempts use different random seeds but identical system prompts and temperature T=1.0, with Python tools executing in a persistent Jupyter sandbox.
- Final answers use weighted majority voting, giving more weight to low-entropy, more confident attempts.
- The time budget reserves 540 seconds for infrastructure variance and 360 seconds for startup before dividing the remaining solving time equally among problems.
3 Diverse Prompt Mixer
Diverse Prompt Mixer tests several reasoning strategies and ensemble configurations while holding other parameters constant. Increasing prompt diversity consistently hurts performance, including a seemingly promising Code-first strategy that fails in confirmation runs.
- Four system prompts are compared: Original, Small Cases First, Work Backwards, and Classify Then Solve.
- The experiments use different strategies while keeping all other parameters identical.
- Code-first scores 41, 38, and 34 across three runs, for a mean of 37.7 below baseline.
- Replacing Original prompts with diverse strategies never helps and eventually hurts performance.
4 Why It Fails
High-temperature sampling already supplies substantial error decorrelation, while prompt diversification reduces accuracy and cannot exploit additional correlation headroom. Across experiments, model capability remains more important than prompt-level optimization, and extra sampling can backfire under fixed compute.
- Temperature Already Decorrelates: T=1.0 is optimal on gpt-oss-120b, while both lower and higher temperatures degrade performance.
- Pairwise Correlation Is Already Near Zero: All 19 computable correlation estimates are negative, with mean ˆρ = −0.122 for N≥7, leaving no correlation headroom for diversity strategies.The estimates use correct-vote counts and ˆp = vc/N, excluding problems with ˆp ∈{0, 1}.
- Prompt Mixer Test: 36 points for the equal prompt mix at T=0.5 is worse than 38 for T=0.5 alone, even when temperature-based diversity is suppressed.
- Model Capability Dominates: At equal N=8, gpt-oss-120b scores 39.3 versus 31.0 for gpt-oss-20b, an 8-point gap that dwarfs prompt optimizations of ±2 points.Scaling gpt-oss-20b to N=32 lowers its score to 26 as per-attempt time shrinks and ˆp drops from 0.61 to 0.52.
- Model Capability Dominates: No inference-time optimization improves over baseline within a fixed model, and the model capability gap persists regardless of compute budget.
5 Cross-Model Validation
Cross-model tests show that prompt interventions do not improve the Qwen baseline, while Nemotron matches its lower leaderboard score despite more active parameters. These results support testing capability across models, although the Nemotron comparison uses fewer attempts.
- Qwen3.5-35B-A3B: 23/50: Qwen3.5-35B-A3B’s leaderboard submission, despite eight controlled experiments and no improvement from doubling N.Long prompts and manufacturer-recommended parameters each reduced performance or caused a crash.
- Nemotron-Super-120B-NVFP4: 23/50: Nemotron-Super-120B-NVFP4’s leaderboard score, identical to Qwen despite four times the active parameters.Its local configurations scored 6–9/10, but the comparison is confounded by N=3 versus gpt-oss-120b’s N=8.
- Qwen3.5-35B-A3B: 8/10: Qwen’s baseline local score, while underperforming interventions reached 7/10 and some configurations crashed.Nothing in the ablation exceeded the baseline.
6 Complete Ablation
The complete gpt-oss-120b ablation finds a flat optimization landscape: no tested intervention reliably beats baseline, and a promising single run fails confirmation. The broader comparison likewise shows no reliable improvement.
- Complete Ablation: 23 experiments: each changed one variable on gpt-oss-120b, and none reliably improved over baseline.This establishes the breadth of the complete ablation.
- Complete Ablation: 41/50: E12 Code-first’s initial score, but confirmation runs fell to 38 and 34, producing a 3-run mean of 37.7 versus the 39.3 baseline mean.The Formalize-First prompt scored 39, and the original system is described as a local optimum.
- Complete Ablation: No experiment reliably exceeds the 39.3 baseline mean in the complete ablation.The table summary also records that E12’s initial 41 was not replicated.
7 Comparison with State of the Art
Competition progression is characterized as a shift from high-N voting toward high per-attempt success probability, with inference-time optimization yielding diminishing returns as models improve. The work reports a 42/50 score while noting a 46+ top score and remaining room.
- AIMO Competition Progression: High-N gives way to high-ˆp: the paper contrasts AIMO-1’s N=48 voting and AIMO-2’s trained GenSelect with this system’s zero training compute.The stated conclusion is “model capability first, everything else second.”
- AIMO Competition Progression: 42/50: this work’s AIMO competition score, compared with winner or top leaderboard scores progressing from 29 to 34 to 46.The figure frames this as a transition from high-N voting toward higher capability.
- Inference-Time Scaling: At ˆp ≈0.69 on IMO-level problems, repeated sampling works, but prompt diversity, temperature tuning, and strategy mixing do not improve vanilla self-consistency.Inference-time optimization returns flatten once the base system is configured.
- Comparison with State of the Art: 39.3 mean and 42 best: the gpt-oss-120b baseline narrows the gap with earlier open-source results, while the AIMO-3 top score of 46+ indicates further room.The cited comparison includes a highest open-source Kaggle score of 34/50.
8 Submission as Lottery
Submission outcomes vary substantially across baseline runs, making each submission a lottery ticket. The mixer lowers the expected score without improving the chance of reaching 42, while infrastructure noise remains a source of variance.
- Submission as Lottery: 39.3 mean, σ=1.7: 21 baseline runs ranged from 34 to 42, with each run having approximately 5.6% probability of scoring at least 42.The best run reached 42/50.
- Submission as Lottery: µ≈39.0: the Mixer’s expected score fell from the baseline while its tail probability did not improve.The corresponding figure reports baseline p≈0.056 and Mixer p≈0.037 per run for reaching at least 42.
- Submission as Lottery: ∼6 percentage points: shared-GPU resource contention alone can shift benchmark results by this amount.The paper states that 21 baseline runs and three-model cross-validation mitigate this variance.
9 Selection Loss
The best majority-vote score trails pass@20 because correct answers can appear in the sampled pool yet lose the vote. Thus, the negative result applies to prompt-level optimization within a fixed selector, while selection-level optimization remains open.
- 45.5 pass@20 versus 42 majority-vote score leaves a six-point selection gap.The correct answer may be present among N=8 attempts but outvoted by a more common wrong answer.
- A verifier-based selector could recover some missed correct answers without ground-truth labels.Suggested checks include code execution against constraints, formal substitution, and cross-candidate consistency.
- The negative finding is limited to prompt-level interventions inside a fixed majority-voting selector.Selection-level optimization is explicitly left as a separate open question.
10 Conclusion
Across the tested AIMO 3 settings, Diverse Prompt Mixer and other prompt-level interventions did not improve performance. The conclusion favors high-temperature sampling and larger-capability models under fixed competition compute constraints.
- 10 Conclusion: Across 3 models, 23+ experiments, and 50 IMO-level problems, model capability dominates prompt-level inference-time optimizations.The reported capability difference is 8 points, compared with prompt effects of approximately ±2 points at equal N=8.
- 10 Conclusion: High-temperature sampling already provides sufficient diversity, making structured prompt diversity redundant at best and harmful at worst.The tested strategy changes included distinct reasoning prompts and other prompt-level interventions.
- 10 Conclusion: For hardware-constrained competitions, the paper recommends using the largest model that fits and keeping temperature high.
E Baseline Score Distribution
The baseline consists of repeated identical runs under a fixed configuration, producing a score distribution centered near 39.3. Reproduction uses the same inference setup and reports roughly 4.5 hours of runtime.
- Twenty-one baseline runs produced a mean score of 39.3, standard deviation 1.7, and range 34–42.
- Expected results are scores from 34–42 per run, with total runtime of approximately 4.5 hours.
- Each reproduction changes only the system prompt or the specified parameter while keeping the remaining configuration identical.The guide describes a fixed model and server setup for replicating experiments.