PaperlayerView exact revision on arXiv

Source-linked AI summary

Making, not Taking, the Best of N

Ammar Khairi, Daniel D'souza, Marzieh Fadaee, Julia Kreutzer

arXiv:2510.00931v1cs.CL

TL;DR

Best-of-N selection discards diverse information from candidate generations, although those candidates may contain complementary strengths. FusioN uses a general LLM judge to synthesize them, and the paper reports it consistently outperforms BoN across test-time scaling and synthetic-data generation.

  • Problem

    Best-of-N selects one candidate and discards the diverse reasoning paths and potentially useful information in the remaining generations.

    Source: §1 · p. 2

  • Method

    FusioN uses a strong generative LLM as a fusor to integrate complementary strengths from multiple candidate generations into one answer.

    Source: §1 · p. 2

  • Results

    FusioN consistently outperforms BoN across test-time scaling and synthetic-data generation, including +3.8% win-rate vs Gemini2.5-Pro on mArena-v2.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings support treating candidate generations as collaborators whose complementary strengths can be synthesized, rather than as competitors from which one winner is selected.

    Source: §1 · p. 2, §7 · p. 12

    The authors have not tested truly adversarial teacher inputs and rely on users to verify teacher suitability.

    Source: Ethics Statement · p. 13

Abstract

from arXiv · show

Obtaining high-quality generations in modern LLMs has largely been framed as a selection problem: identifying a single winning generation from a diverse pool of N samples, the Best-of-N (BoN). Yet, this approach is inherently zero-sum, discarding diverse and potentially useful information from the pool. Instead, we explore a collaborative setup, where all candidates can potentially contribute to the final winning generation. To this end, we propose Fusion-of-N (FusioN): a method that uses a general LLM judge to synthesize the most informative elements of each sample into a single final answer. We compare FusioN to BoN in two settings, (i) test-time scaling, where we sample and aggregate from a single model at test-time (ii) synthetic data generation, where we fuse samples from a pool of diverse teachers to improve a student model. We extensively benchmark both setups across 11 languages, 3 diverse tasks and varying model scales. Across the bench, FusioN consistently outperforms BoN showing versatility and robustness both in test-time scaling and in downstream gains from synthetic data generation. We also perform extensive analysis on FusioN, where it shows surprising strengths and robustness under challenging settings. These results show that we should shift how we think about evaluating and utilizing LLM generations from a monolithic measure of quality, to embracing their polylithic nature. This shift allows us to integrate diverse strengths, unlock latent potential, and achieve improvements that were previously inaccessible through selection alone.

1 Introduction

Best-of-N selection discards potentially useful diversity, prompting FusioN to synthesize complementary candidate strengths instead. Across test-time scaling and synthetic-data generation, FusioN outperforms BoN and shows sample efficiency and robustness.

The problem with selection

  • BoN discards diverse reasoning paths, wastes sampling compute, and can favor a high-scoring candidate that is less correct or useful.The paper identifies this judge-score mismatch as a risk of reward hacking.Source: §1 · p. 2

Conceptual shift from selection to synthesis

  • FusioN uses a strong LLM judge to integrate complementary candidate insights into one answer, rather than selecting a single generation.The paper presents this synthesis approach as a simple alternative to BoN that can work out of the box with a reasonably strong generative LLM as fusor.Source: §1 · p. 2

Evaluation scope

  • Evaluations cover test-time scaling and synthetic-data fine-tuning across multiple languages, with open-ended generation, translation, math, and factual reasoning tasks.The evaluation uses multiple languages in both settings to test FusioN under diverse and challenging conditions.Source: §1 · p. 2

Demonstrated gains across test-time scaling and data generation

  • +3.8% win-rate vs Gemini2.5-Pro on mArena-v2 and +3.7 XCometXL on translation accompany FusioN’s gains across test-time scaling and synthetic-data generation.Synthetic-data gains include +2.5% on mArena-v2 vs Gemini2.5-Flash, +0.8 on WMT, +1.8% on GeoFactX answer accuracy, and +1.1% on reasoning quality.Source: §1 · p. 3

Robustness and efficiency across models and settings

  • FusioN is more sample-efficient and robust than BoN, including with smaller or weaker teacher pools, but has limits on tightly constrained math tasks.The paper also reports that FusioN data can help fine-tuned models outperform the strongest single teacher.Source: §1 · p. 3

2 Methodology: From Selection to Synthesis

BoN selects the candidate with the highest scalar score, while FusioN has a generative fusor synthesize a new response from the prompt and candidate pool. Selection is bounded by its pool and scoring alignment; synthesis can combine useful parts across candidates.

Selection with Best-of-N (BoN)

  • BoN selects the candidate with the maximum scalar score, using a scoring function that may come from a reward model or a prompted generative LLM.The paper notes trained classifiers often perform better than generative LLMs at producing scalar scores.Source: §2 · p. 3

Limitations of BoN

  • BoN is limited by scoring alignment and candidate-pool quality, and improving weak pools by scaling sample counts can require thousands of samples.The paper notes that imperfect reward proxies risk overfitting and that such scaling can be extremely resource-intensive.Source: §2 · pp. 3–4

Synthesis with Fusion-of-N (FusioN)

  • A standard LLM fusor generates a new response from the prompt and candidate pool, making the final output conditionally dependent on the other candidates.Unlike BoN, the synthesized response can exceed the original pool in quality.Source: §2 · p. 4
  • FusioN can mix and match high-quality fragments of varying sizes, while retaining the option to copy one whole candidate when it is best overall.The approach treats quality as polylithic: individual samples can contain both higher- and lower-quality parts.Source: §2 · p. 4

Components of FusioN

  • FusioN’s success depends on the fusor’s comparative evaluation and aggregation abilities, and an adequate model-size threshold may be needed without specialized training.Its prompt can be tuned for ambiguous safety, tone, or model-identity preferences, and to control whether weaker parts are discarded.Source: §2 · p. 4

3 Experimental Setup

The experiments compare FusioN and BoN on identical generations, varying the aggregation method. They use multilingual test-time scaling and synthetic-data fine-tuning setups, with benchmarks spanning open-ended generation, translation, and reasoning.

3 Experimental Setup

  • Both aggregation methods receive identical generations for the same prompts and differ in how they produce the final output.The experiments cover test-time scaling and synthetic-data generation.Source: §3 · p. 4

3.1 Models for Test-Time Scaling

  • Aya Expanse 8B and Command A generate five samples at temperature 0.7; a reward model scores BoN candidates and Command A serves as a FusioN fusor.The paper also evaluates variations in sample count and compares fusor choices.Source: §3.1 · pp. 4–5

3.2 Models and Data for Synthetic Data Generation

  • Five open teacher models provide synthetic-data candidates, which are selected by reward model or Command A fusor for fine-tuning 111B and, in an ablation, 7B models.Training prompts include 10,000 UltraFeedback Binarized prompts translated into nine languages, plus GeoFactX and translated s1k reasoning prompts.Source: §3.2 · p. 5

3.3 Evaluation Benchmarks

  • Benchmarks cover 11-language open-ended generation, English-to-10-language translation, and multilingual factual and mathematical reasoning with win-rate, XComet-XL, or accuracy measures.For open-ended generation, FusioN raises win rates against Gemini2.5-Pro across languages and largely outperforms BoN for both evaluated models at N = 5.Source: §3.3 · pp. 5–6, Figure 2 · p. 6

4 Results

FusioN outperforms BoN in test-time scaling and in downstream evaluations of models fine-tuned on fused data. Results span open-ended generation, translation, and multilingual factual reasoning, with some variation across languages and tasks.

4.1 Test-time Scaling

  • FusioN beats BoN in 9/11 languages on open-ended generation and outperforms translation BoN across languages, exceeding the oracle in German, Russian, and Chinese.In translation, the reported margin reaches +11.4 XCometXL in Korean; FusioN scores +0.8 above the oracle in Chinese.Source: §4.1 · pp. 6–7, Figure 3 · p. 6

4.2 Synthetic Data Generation

  • FusioN-fine-tuned models score +0.8 XCometXL on average over BoN-fine-tuned models on WMT24++ and improve win-rates against Gemini2.5-Flash by +2.5%.Table 1 covers 111B models on Arena and WMT across languages.Source: §4.2 · p. 7, Table 1 · p. 7
  • FusioN-fine-tuned models outperform BoN-fine-tuned models in factual reasoning across four of five languages, with a minor regression in Hindi.For Swahili, the reported gain over BoN is +2.8% in answer correctness and reasoning score; gains also hold in fusor-unsupported Swahili and Thai, while MGSM includes cases below BoN.Source: §4.2 · pp. 7–8, Figure 4 · p. 8

5 Analysis

The analyses examine how FusioN responds to fusor scale, test-time budget, and teacher-pool composition. They find gains with larger fusors and small budgets, robustness to weaker pools, and limits on language and task coverage.

Fusor scale

  • FusioN win-rates rise by +5.5% when the fusor scales from 27B to 111B, while BoN’s small-judge advantage vanishes at larger scale.The comparison measures average Arena win-rate across languages.Source: §5 · pp. 8–9, Figure 5 · p. 8

Test-time budget

  • At N = 2, FusioN improves win-rate against Gemini2.5-Pro by +6%, while BoN needs twice as many samples for similar gains.Both methods plateau beyond N = 7; BoN’s independent samples are parallelizable, whereas FusioN encodes all samples together.Source: §5 · p. 9, Figure 6 · p. 9

Teacher pool and data quality

  • FusioN-generated data beats BoN data by +4.4% in the default five-teacher setup and still wins with weaker or smaller pools, though a single-teacher pool lowers win-rates substantially.Replacing a teacher with a weaker one minimally affects both methods; reducing the pool to four teachers affects FusioN proportionally more, but it remains ahead of BoN.Source: §5 · p. 9, Table 2 · p. 10

Teacher contributions

  • FusioN combines substantial teacher-output spans with limited unmatched glue, and on GeoFactX shifts toward Gemma3-27B-It when the fusor lacks language support.Its contributions include even less-preferred teachers in UFB, while teacher preferences shift on GeoFactX languages unsupported by Command A.Source: §5 · pp. 9–10, Figure 7 · p. 10, Figure 8 · p. 10

Fusor quality

  • On GeoFactX, FusioN achieves 58.6% answer accuracy despite Command A scoring lowest, and it also leads on the internal reward score.The authors attribute the 98.9% language-correctness result as likely related to an English-only fusion prompt.Source: §5 · p. 10

Limitations and opportunities

  • FusioN benefits English more than other languages, and BoN may be safer for cross-lingual transfer to lower-resource languages; MGSM also shows mixed results.The authors suggest close-ended tasks may be poorly suited to generative ensembling or may require specialized fusor training.Source: §5 · pp. 10–11

6 Related Work

The paper situates FusioN among ensemble-learning and synthesis-based approaches, emphasizing its single-step, untrained aggregation. It also relates the method to test-time scaling and multilingual synthetic-data generation.

Learning from Ensembles

  • FusioN takes ensemble outputs as input to a fusor LLM, which integrates their strengths into one output, rather than combining model probabilities or training from consensus.The paper frames this as integrative output ensembling.Source: §6 · p. 11

Synthesis-based Ensembling

  • Unlike multi-layer MoA or LLM-Blender’s ranking-plus-fusion pipeline, FusioN works in one aggregation step without training a specialized aggregator.The authors report robustness to teacher pools in their diverse generative settings, while single-teacher sampling performs significantly worse.Source: §6 · pp. 11–12

Test-time Scaling and Synthetic Data Generation

  • FusioN can be viewed as parallel sampling followed by sequential refinement, with adaptive compute proposed as a future direction; multi-teacher synthetic-data generation remains underexplored.Prior multilingual data efforts described here include rejection sampling and routing samples among multiple teachers.Source: §6 · p. 12

7 Conclusion

The authors argue that FusioN outperforms BoN across test-time scaling and synthetic data generation, including when teacher pools contain weaker models. They conclude that integrating diverse strengths can improve on selection alone.

7 Conclusion

  • Across test-time scaling and synthetic data generation, FusioN consistently outperforms winner-takes-all approaches such as BoN.The conclusion also reports higher-quality outputs, greater sample efficiency, and stronger downstream performance.Source: §7 · p. 12
  • FusioN leverages multiple models’ strengths, including when some contributors are weaker, showing robustness and adaptability.Source: §7 · p. 12
  • The authors argue that embracing the polylithic nature of generation quality can integrate diverse strengths and achieve improvements unavailable through selection alone.Source: §7 · p. 12

Ethics Statement

The ethics statement notes risks of bias amplification and adversarial teacher inputs, alongside safeguards and user responsibilities. It also reports a slight position bias in the fusor’s preferences.

Ethics Statement

  • Synthetic-data training risks propagating and amplifying biases; increasing teacher diversity may reduce model-specific bias propagation, but this is not guaranteed.Source: Ethics Statement · p. 13
  • The authors have not tested truly adversarial inputs; users must assess teacher suitability, as prompt injections may be more likely with FusioN than BoN.Source: Ethics Statement · p. 13
  • FusioN instructions can include safety guidelines, and a hard filter could exclude unsafe or lowest-quality samples before aggregation.Source: Ethics Statement · p. 13
  • The fusor showed no evidence of self-preference but did show a slight position bias toward samples presented first.Source: Ethics Statement · p. 13

Reproducibility Statement

This section documents the prompts, training settings, data and evaluation procedures, and reproducibility provisions. It also reports that BoN has a slight but consistent edge over FusioN on most tested MGSM languages.

Reproducibility Statement

  • The authors release synthetic data and pairwise evaluations where licenses allow, but sampling randomness means the data-generation pipeline is not perfectly reproducible.Source: Reproducibility Statement · p. 13

A Prompts

B Fine-tuning Hyperparameters

  • 111B UFB training uses batch 16, Adam, 5e-6 peak learning rate, 64 H100s, and 250 steps; the extended mix uses 323 steps, while 7B uses 16 GPUs.Source: B Fine-tuning Hyperparameters · p. 25

C Synthetic Reasoning Data

  • They use translated prompts and teacher-generated responses; GeoFactX human-verified answers support accuracy evaluation, while s1k-X silver-answer formats hinder correctness analysis.Source: C Synthetic Reasoning Data · pp. 25, 27

D Evaluation

  • Evaluation covers multilingual open-ended prompts, English-to-10-language translation, 11-language grade-school math, and GeoFactX reasoning and factual answers.Source: D Evaluation · pp. 27–28

E.1 Test-time Scaling

  • On MGSM, BoN has a slight but consistent edge over FusioN across seven languages, except German; FusioN remains relatively close to Oracle performance.Source: §E.1 · p. 28, Figure 9 · p. 28

E.2 Synthetic Data Generation

In downstream synthetic-data results, FusioN performs best on GeoFactX, while BoN and the fusor judge outperform it on MGSM. The authors note that prior training overlap may complicate interpretation of the math results.

E.2 Synthetic Data Generation

  • On GeoFactX, the FusioN-trained model leads BoN by 1.0% and the fusor judge by 4.8%.Source: Figure 10 · p. 29, §E.2 · p. 29
  • On MGSM, FusioN trails BoN by 1.2% and the judge by 1.5%, yet improves 3.6% over Base.Source: §E.2 · pp. 28–29, Figure 10 · p. 29
  • The authors suggest popular s1k data may already have appeared in prior training, potentially complicating interpretation of the MGSM results; they contrast this with clearly unseen GeoFactX data.Source: §E.2 · p. 29

F Contribution Analysis

FusioN’s contribution analysis measures how teacher outputs and the fusor contribute to fused answers, then probes self-bias, candidate-model effects, and sample-order effects.

Measuring contributions

  • The analysis uses string matching to find contiguous teacher-output blocks, assigns characters to the teacher with the longest match, and counts attributed characters.Ties are labeled “multiple,” while characters with no match are “unmatched”; matching may miss semantic rephrasings.Source: F Contribution Analysis · p. 29

Disentangling fusor and teacher pool

  • With the fusor excluded, FusioN and BoN both favored DeepSeek-V3, which the authors take as evidence against fusor self-bias.The comparison used a small UFB subset in Arabic, English, and Chinese.Source: F Contribution Analysis · p. 30, Figure 11 · p. 30

Contributions in test-time scaling

  • For both Aya Expanse 8B and Command A candidates, most fused content came from samples, with unmatched content remaining small.The unmatched rate was larger for Aya Expanse 8B; outputs were almost always more than 80% sample content.Source: F Contribution Analysis · p. 30, Figure 12 · p. 30

Position bias

  • Contribution ratios were highest for the first and second samples across three languages, although the reported position bias was not large.Samples were shuffled before prompt formatting, and order was measured as seen by the fusor.Source: F Contribution Analysis · pp. 30–31

G Evaluation Results Comparison

Head-to-head tests favor FusioN over BoN in test-time scaling, while fine-tuning comparisons vary across model sizes and tasks. The usage disclosure lists several auxiliary AI uses.

G Evaluation Results Comparison

  • For 7B fine-tuning, FusioN improved WMT24++ in every language but showed no significant gain over BoN on m-ArenaHard-v2.0.The authors suggest that this smaller model may require more parameter tuning, particularly given the relatively small synthetic dataset.Source: G Evaluation Results Comparison · p. 31
  • For the 111B model’s multilingual factual-reasoning evaluation, fine-tuning improved over baseline except in Hindi, and FusioN beat the fusor in most results.The baseline was the model used for fine-tuning; the fusor generated the fused data.Source: G Evaluation Results Comparison · p. 33
  • FusioN outperformed BoN across languages in head-to-head test-time scaling, with larger gains for Aya Expanse 8B than Command A.The comparison used five Arena samples per prompt and GPT-4o judging; the reported Command A margin reached +55.2% win-rate in Italian.Source: G Evaluation Results Comparison · p. 33, Figure 15 · p. 35
  • For 7B fine-tuned models, BoN scored better across all languages in direct comparison, while 111B results varied by language.The corresponding figure reports that the variants were mostly on par at 111B, while BoN was better at 7B.Source: G Evaluation Results Comparison · p. 33, Figure 16 · p. 35
  • FusioN generally grew faster than BoN across four languages in scaling plots, with the size of the advantage depending on language.The plots report win rates against Gemini2.5-Pro on m-ArenaHard-v2.0.Source: G Evaluation Results Comparison · p. 33, Figure 17 · p. 35

H LLM Usage Disclosure

  • The usage disclosure lists AI assistance for table formatting, shortening text, finding English correspondences, contribution-analysis implementation, and expanding the related-work list.The authors say the related works were then read and curated into the final discussion.Source: H LLM Usage Disclosure · p. 33
Loading 2510.00931v1…