PaperlayerView exact revision on arXiv

Source-linked AI summary

When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling

Heecheol Yun, Kwangmin Ki, Junghyun Lee, Eunho Yang

arXiv:2510.15346v2cs.CLcs.AI

TL;DR

Long-form LLM ensembling remains underexplored, and ensembling at every token can degrade performance. SAFE chooses token positions using tokenization mismatch and model consensus, and experiments show it outperforms existing methods in accuracy and efficiency. Its reported advantages include stability and reduced computational cost, though it does not always beat the best individual model and experiments are limited to non-reasoning models.

  • Problem

    Probability-level ensembling has shown effectiveness for short-form answers, but its application to long-form generation remains underexplored and requires careful choice of ensembling positions.

    Source: §1 · p. 1

  • Method

    SAFE selectively ensembles at token positions chosen by considering tokenization mismatch and consensus in models’ next-token distributions, with probability sharpening when the ensemble distribution is overly smooth.

    Source: §1 · pp. 2–3, §3.3 · p. 7

  • Results

    SAFE outperforms existing methods in stability and computational efficiency, and consistently improves recent ensembling approaches across diverse model combinations.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    For long-sequence generation, ensembling at appropriate token positions can improve accuracy and efficiency compared with ensembling at every token.

    Source: §1 · pp. 1, 3

    SAFE does not always outperform the best individual model, and the experiments are limited to non-reasoning models.

    Source: B LIMITATIONS AND FUTURE WORK · p. 16

Abstract

from arXiv · show

Ensembling Large Language Models (LLMs) has gained attention as a promising approach to surpass the performance of individual models by leveraging their complementary strengths. In particular, aggregating models' next-token probability distributions to select the next token has been shown to be effective in various tasks. However, while successful for short-form answers, its application to long-form generation remains underexplored. In this paper, we show that using existing ensemble methods in long-form generation requires a careful choice of ensembling positions, since the standard practice of ensembling at every token often degrades performance. We identify two key factors for determining the ensembling positions: tokenization mismatch across models and consensus in their next-token probability distributions. Based on this, we propose SAFE, (Stable And Fast LLM Ensembling), a framework that selectively ensembles by jointly considering these factors. To further improve stability, we apply a probability sharpening strategy when the ensemble distribution becomes overly smooth, enabling the selection of more confident tokens during ensembling. Our experiments on diverse benchmarks, including MATH500 and BBH, demonstrate that SAFE outperforms existing methods in both accuracy and efficiency, with gains achieved even when ensembling fewer than 1% of tokens.

1 INTRODUCTION

In long-form generation, ensembling effectiveness depends on selecting appropriate token positions based on tokenization mismatch and model consensus. SAFE selectively ensembles at those positions to improve stability and efficiency.

1 INTRODUCTION

  • Tokenization mismatches can create OOV-like prefixes that corrupt a model’s next-token distribution and propagate errors during long-sequence generation.For example, “So” is an unnatural prefix for a model that tokenizes “Sofia” as one token.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • When models’ next-token distributions sufficiently agree, the most confident token can be identified without aligning their vocabularies, saving ensemble operations.Vocabulary alignment is costly because it maps across large vocabulary sets.Source: §1 · p. 2
  • SAFE selects ensemble positions using tokenization mismatch and model consensus, and applies a Generate–Verify–Ensemble cycle to validate and replace selected drafter tokens.When the ensemble distribution is overly smooth, SAFE sharpens it to concentrate probability mass on more plausible tokens.Source: §1 · pp. 2–3
  • Ensembling at every token can degrade CoT performance, whereas direct multiple-choice answering avoids the tokenizer mismatches described for long-form generation.The supplied Table 1 caption reports this contrast for UniTE.Source: §1 · p. 2, Table 1 · p. 3
  • SAFE reduces computation, supports stable generation, and consistently improves recent ensembling approaches across diverse model combinations.Its speculative strategy limits autoregressive generation to one drafter, while selective ensembling reduces ensemble operations.Source: §1 · p. 3

2 RELATED WORK

Related work covers response-level and token-level LLM ensembling, probability aggregation across heterogeneous vocabularies, and speculative decoding. The paper identifies long-sequence reasoning with different tokenizers as a challenge for these approaches.

2.1 LLM ENSEMBLE

  • LLM ensembling is organized by whether aggregation happens after inference or during inference.The paper reviews both directions of work.Source: §2.1 · p. 3

2.1.1 ENSEMBLE AFTER INFERENCE

  • After-inference methods include cascade, parallel, and hybrid structures for selecting or combining complete model responses.The cited approaches use cost-based routing, response ranking, or aggregation, and can require numerous calls.Source: §2.1.1 · pp. 3–4

2.1.2 ENSEMBLE DURING INFERENCE

  • During-inference methods include token routing and probability-level aggregation, which aligns models’ heterogeneous vocabularies but faces stability and efficiency challenges in long reasoning sequences.OOV-like tokens and repeated autoregressive generation contribute to those challenges.Source: §2.1.2 · p. 4

2.2 SPECULATIVE DECODING

  • Speculative decoding uses a small drafter to propose tokens that a larger target model checks in one forward pass, reducing autoregressive generation cost.Prior speculative probability-level ensembling is limited to shared tokenizers; the paper proposes an acceptance criterion for heterogeneous tokenizers.Source: §2.2 · p. 4

3 SAFE: TOWARD STABLE AND FAST LLM ENSEMBLING

SAFE uses a drafter to generate lookahead tokens and verifiers to select where ensembling is needed, jointly accounting for tokenizer mismatch and model consensus. It combines selective ensembling with probability sharpening when distributions are overly smooth.

3 SAFE: TOWARD STABLE AND FAST LLM ENSEMBLING

  • SAFE selects ensemble points by jointly considering tokenizer mismatch and next-token distribution consensus, with verifiers guiding a drafter’s lookahead generation.At selected points, the drafter’s token is replaced using the ensemble distribution.Source: §3 · p. 5, Figure 3 · p. 5

3.1 GENERATE

  • The drafter generates a lookahead sequence so SAFE can account for tokenization schemes that represent the same word with different token boundaries.For example, one model may tokenize “Incorrect” as three tokens while another uses one.Source: §3.1 · p. 5

3.2 VERIFY

  • Verifiers skip ensembling after OOV-like tokens, whose prefix boundaries do not match a verifier’s tokenization, to avoid conditioning models on invalid prefixes.A token is OOV-like for a verifier when the decoded prefix boundary does not match any boundary in that verifier’s tokenization.Source: §3.2 · pp. 5–6
  • For tokens passing the boundary check, SAFE skips ensembling when model-level checks indicate the token is already the most confident choice.These checks avoid repeatedly constructing the exact ensemble distribution; the paper states they do not compromise accuracy compared with using it.Source: §3.2 · pp. 5–6

3.3 ENSEMBLE: SHARPENING ENSEMBLE DISTRIBUTION

  • When the ensemble distribution is overly smooth, SAFE sharpens it by reallocating mass toward common-prefix tokens or using a geometric mean.The heuristic reallocation applies only to drafter tokens whose initial probability exceeds λ; the geometric mean concentrates mass on tokens consistently supported across models.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

The experiments evaluate SAFE across models with heterogeneous and highly similar tokenizers, diverse benchmarks, and ensemble baselines. Results show that selective ensembling can improve performance while reducing ensemble operations and maintaining latency close to individual models.

4.1 EXPERIMENTAL SETTINGS

  • The experiments use five benchmarks spanning general knowledge, mathematics, and reasoning, with zero-shot CoT except for 3-shot CoT on BBH.They compare three models with 40%–60% tokenization agreement and two models with more than 99% agreement, and evaluate GaC and UniTE.Source: §4.1 · p. 7

4.3 MAIN ANALYSIS

  • SAFE lets UniTE achieve the best performance in 9/15 cases while reducing ensembling to fewer than 20% of tokens on heterogeneous-tokenizer experiments.The reported comparison uses CoT and E/T denotes the percentage of tokens undergoing ensembling.Source: Table 2 · p. 8, §4.3 · p. 8
  • SAFE triggers ensembling for 4.85% of tokens on average in math datasets, versus 15.24% in general-domain datasets.The paper attributes the difference to higher verifier agreement on structured math responses and greater linguistic variability in general-domain responses.Source: §4.3 · pp. 8–9
  • Three-model ensembling does not consistently outperform two-model ensembling, and selecting the top two models typically yields the strongest results.When model rankings are unknown, ensembling multiple comparable models provides stable, though not necessarily optimal, performance.Source: §4.3 · p. 9
  • SAFE closely matches individual-model latency over long sequences and improves long-response efficiency through drafter-only autoregression, fewer ensemble operations, and KV caching.The KV-cache comparison appears in Figure 5(c), whose time-axis scale differs from panels (a) and (b).Source: §4.2 · p. 8, §4.3 · p. 9, Figure 5 · p. 9
  • With highly similar tokenizers, applying SAFE to existing methods yields over a 9% gain on MATH500 compared with the best-performing individual model.The evaluated model pairs tokenize more than 99% of Oxford 5000 words identically.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 ABLATION STUDY

  • Probability sharpening consistently improves performance across benchmarks and ensemble methods, while multiple sharpening strategies offer gains and overly high heuristic thresholds reduce effectiveness.The geometric mean generally performs strongly; the authors leave the sharpening method as a flexible design choice.Source: §4.4 · pp. 9–10, Table 4 · p. 9, Table 5 · p. 10
  • A drafter sequence length of 5 provides the best balance between accuracy and efficiency; shorter sequences may slightly reduce performance, while longer ones may reduce efficiency.The table caption identifies the sequence-length variants as SAFEn, where n is the number of generated tokens.Source: §4.4 · p. 10, Table 6 · p. 10

5 CONCLUSION

The paper finds that choosing when to ensemble is critical to accuracy and efficiency in long-form generation. SAFE uses tokenization mismatch and model consensus to select ensemble points, with probability sharpening and KV-cache management supporting accuracy and speed.

5 CONCLUSION

  • Across widely used 7B-scale model combinations, SAFE outperforms existing methods with only a few ensemble operations.The conclusion describes SAFE as a generate-verify-ensemble framework and notes that KV-cache implementation enables much faster ensembling in long-form generation.Source: §5 · p. 10

A EXPERIMENTAL DETAILS

The experimental details specify prompts and BBH subject selection, and explain the choice of probability-level ensemble baselines.

A EXPERIMENTAL DETAILS

  • The BBH evaluation filters to 15 subjects where the models exhibit comparable performance, following the observation that ensembling is meaningful for similarly performing base models.The selected subjects are listed in the paper; multiple-choice and math prompts use separate templates.Source: A EXPERIMENTAL DETAILS · p. 16, Figure 6 · p. 16, Figure 7 · p. 16
  • The baseline comparison uses GaC, which ensembles below a 0.5 main-model probability, and UniTE, a state-of-the-art method that ensembles at every generation step.The paper applies SAFE to both baselines to assess its versatility in deciding when to ensemble.Source: A EXPERIMENTAL DETAILS · p. 16

B LIMITATIONS AND FUTURE WORK

SAFE reduces latency on general-domain tasks while maintaining speeds comparable to individual models on long sequences. The additional sections describe its performance boundaries, an accuracy-preserving verification criterion, and KV-cache alignment.

B LIMITATIONS AND FUTURE WORK

  • SAFE can underperform the strongest individual model in some cases; its reported general advantage is for ensembles of similarly capable models, and experiments use non-reasoning models.The authors attribute some underperformance to weaker models elevating an incorrect token in the ensemble distribution.Source: B LIMITATIONS AND FUTURE WORK · p. 16

C CORRECTNESS OF ENSEMBLE DISTRIBUTION VERIFICATION

  • Under its stated verifier conditions, Theorem 1 says ensemble-distribution verification preserves the ensemble-selected token without compromising accuracy.The conditions concern verifier consensus or average probability.Source: C CORRECTNESS OF ENSEMBLE DISTRIBUTION… · p. 17

D DETAILS OF OUR KV CACHE MANAGEMENT

  • SAFE prunes each participating model’s KV cache after ensemble steps so cached states match the actual input sequence before generation continues.The pruning addresses cases where the ensemble output differs from the drafter’s generated tokens.Source: Figure 8 · p. 17, D DETAILS OF OUR KV CACHE MANAGEMENT · p. 18

E LATENCY COMPARISON

  • SAFE substantially reduces MMLU-redux latency versus existing ensemble methods and reaches speeds comparable to individual models on hundreds-token generations.Figure 9 compares latency; its panel (c) uses a different time-axis scale from (a) and (b).Source: Figure 9 · p. 18, E LATENCY COMPARISON · p. 18

F ABLATION ON THE DRAFTER’S SEQUENCE LENGTH

Figure 10 compares latency across drafter sequence lengths on MMLU-redux; longer sequences reduce efficiency.

F ABLATION ON THE DRAFTER’S SEQUENCE LENGTH

  • Longer drafter sequences reduce SAFE_n efficiency in the MMLU-redux latency comparison.Figure 10 varies the drafter’s sequence length, denoted n.Source: Figure 10 · p. 18

G ENSEMBLING LARGER MODELS

The appendix reports SAFE’s performance with 32B-scale models, describes relaxed acceptance thresholds, and clarifies how the paper identifies OOV-like tokens relative to non-canonical tokenization.

G ENSEMBLING LARGER MODELS

  • On 32B-scale models evaluated on MMLU-redux and MATH500, applying SAFE to existing ensemble methods consistently outperformed the baselines.MMLU-redux* excludes nine subjects where Qwen2.5-32B-Instruct largely outperforms EXAONE-3.5-32B-Instruct by more than 10%.Source: G ENSEMBLING LARGER MODELS · pp. 18–19, Table 7 · p. 19

H RELAXED ACCEPTANCE THRESHOLD

  • Relaxed acceptance thresholds support weighted probability combinations and speculative sampling; sampling omits the unanimous-verifier condition and relaxes the second condition.For weighted sums, the second condition is adjusted; for speculative sampling, skipping ensembling is probabilistic.Source: H RELAXED ACCEPTANCE THRESHOLD · p. 19

I FURTHER DISCUSSION OF OOV-LIKE TOKENS

  • OOV-like tokens are identified by checking whether drafter token boundaries align with verifier tokenization boundaries.They are tokens that can corrupt a participating model’s next-token distribution because of cross-model tokenization differences.Source: I FURTHER DISCUSSION OF OOV-LIKE TOKENS · p. 19
  • Unlike non-canonical tokenization, OOV-like cases concern only specific mismatched tokens and tokenizations used by participating models, not every possible segmentation.For example, if participating models tokenize “Hello” identically, the paper identifies no OOV-like token even though other segmentations are possible.Source: I FURTHER DISCUSSION OF OOV-LIKE TOKENS · pp. 19–20

J COMPARISON WITH METHODS THAT ENSEMBLING AT A SPAN-LEVEL

SAFE addresses tokenization mismatch within token-level ensembling, whereas span-level methods avoid the mismatch by ensembling only across larger units.

J COMPARISON WITH METHODS THAT ENSEMBLING AT A SPAN-LEVEL

  • Span-level ensembling avoids tokenization mismatch but cannot ensemble inside a span, sacrificing token-level probability information; SAFE instead selects mismatch-free drafter-token positions.This lets SAFE ensemble within a span while avoiding OOV-like issues.Source: J COMPARISON WITH METHODS THAT… · p. 20

K QUALITATIVE RESULTS

SAFE’s qualitative examples show it can produce correct solutions when individual models fail, while existing ensemble methods can generate erroneous tokens or reasoning that leads to wrong answers.

K QUALITATIVE RESULTS

  • Table 8 shows SAFE yielding the correct solution even when individual models fail to produce the correct answer or reasoning.Source: K QUALITATIVE RESULTS · p. 20
  • Tables 8 and 11 illustrate existing ensemble methods repeatedly generating erroneous tokens after introducing OOV-like tokens.Source: K QUALITATIVE RESULTS · p. 20
  • Tables 9 and 10 show existing ensemble methods producing incorrect reasoning that leads to wrong answers.Source: K QUALITATIVE RESULTS · p. 20
Loading 2510.15346v2…