PaperlayerView exact revision on arXiv

Source-linked AI summary

Funnel of Thoughts: Efficient Test-Time Scaling via Early Voting and Rollout Pruning

Chanhee Park, Sungbin Han, Jeongho Yoon, Seongtae Hong, Heuiseok Lim

arXiv:2608.15065v1cs.AI

TL;DR

Multi-sample voting improves reasoning reliability but requires every long rollout to finish, making attention computation expensive. Funnel of Thoughts prunes hesitation-heavy trajectories and uses early voting while preserving the full 32-rollout accuracy at roughly half the attention FLOPs.

  • Problem

    Majority voting improves multi-sample reasoning reliability, but completing every long rollout makes attention computation prohibitively expensive.

  • Method

    Funnel of Thoughts runs the full rollout pool, banks committed answers, and prunes active trajectories with high hesitation-marker density using generated text.

  • Results

    FoT matches SC@32 accuracy at roughly half the attention FLOPs and reduces full-model FLOPs by 28.8%.

  • Takeaways & Limitations

    Hesitation density is more reliable for identifying disproportionately erroneous rollout tails than predicting individual rollout correctness.

  • Takeaways & Limitations

    The evaluation focuses on reasoning tasks with extractable final answers, leaving open-ended generation and tasks without answer-extraction correctness outside scope.

Abstract

from arXiv · show

Large Reasoning Models produce diverse, sometimes inconsistent answers across repeated queries on the same problem, so multi-sample inference is a prerequisite for reliable deployment. Majority voting at k rollouts is the standard solution and the de facto accuracy target for this regime, but it is prohibitively expensive at the scale LRMs require. We introduce Funnel of Thoughts (FoT), an inference-time method that preserves the full 32-trajectory voted accuracy while halving its attention FLOPs, a 28.8% reduction in full-model inference cost. Across 115K reasoning trajectories from six LRMs, we find that unproductive trajectories often reveal themselves through repeated hesitation markers such as "Wait", "Actually", and "perhaps." These trajectories are less likely to reach the correct answer and consume disproportionate attention FLOPs, degenerating into no-answer loops in the worst case. Built on this training-free lexical signal, FoT identifies the vocabulary that captures these pathological patterns and prunes affected trajectories before completion, reducing online generation attention FLOPs by 56.1% and wall time by 37.6% without any additional model inference; the same signal transfers without retuning across held-out architectures and out-of-domain tasks.

1 Introduction

Large Reasoning Models generate diverse, sometimes inconsistent trajectories, making multi-sample majority voting valuable but expensive because long rollouts must run to completion. Funnel of Thoughts (FoT) addresses this inefficiency by pruning unproductive trajectories in flight using hesitation markers, while preserving the productive trajectories that carry the vote and transferring unchanged across models and tasks.

  • Motivation: LRMs produce diverse, sometimes inconsistent answers across repeated queries, so multi-sample inference can recover correct answers absent from any single rollout.The correct answer often appears somewhere in the rollout pool even when no individual rollout is reliable.
  • Motivation: Self-Consistency samples k completed rollouts and returns the majority vote, but long LRM trajectories make every additional token increasingly expensive.Attention cost grows with the square of sequence length, and rollouts can run to thousands of tokens.
  • Funnel of Thoughts: FoT runs a full pool of k parallel trajectories and truncates only spiraling minority rollouts, retaining productive trajectories that carry the vote.Unlike efficient-sampling methods that draw fewer trajectories, FoT removes unproductive ones in flight using only generated text.
  • Funnel of Thoughts: Hesitation markers such as “Wait,” “Actually,” and “perhaps” occur significantly more often in incorrect trajectories and provide a zero-cost signal for identifying unproductive reasoning.The signal was analyzed across 115K rollouts.
  • Contributions: FoT prunes rollouts at inference time in parallel batches without training, reward models, or logit access, and transfers unchanged across held-out architectures and out-of-domain tasks.The method preserves SC@32 accuracy at substantially reduced compute.

2 Related Work

Prior work identifies substantial inefficiency in reasoning trajectories and develops interventions at the single-rollout, sampling, and rollout-pool levels. Test-time scaling motivates these methods, while verifier-based selection and budget forcing introduce additional cost or different lexical mechanisms.

  • Trajectory inefficiency: 48% of generated tokens on MATH500 are spent on redundant operations such as repetitive self-verification and unnecessary case exploration.Su et al. also report a non-monotonic relationship between trajectory length and accuracy, with both very short and very long rollouts underperforming.
  • Single-trajectory interventions: Single-trajectory methods suppress unproductive reasoning through logit penalties, keyword suppression, or hidden-state-based dynamic termination.TIP penalizes excessive thought-switching, Wang et al. suppress self-reflection keywords, and Wu et al. use hidden-state probes to terminate thinking.
  • Sampling-cost reduction: Majority voting samples k reasoning paths for large gains but requires k full-length trajectories, motivating sequential stopping methods.Adaptive Consistency uses a Beta stopping rule, Certaindex uses a learned stability metric, and other work further refines stopping criteria.
  • Rollout-pool methods: Parallel rollout-pool methods pursue different objectives: Slim-SC prunes by embedding similarity, whereas CISC reweights votes after completion using token-level confidence.Slim-SC requires a separate embedding model; CISC requires logprob access and does not reduce generation cost.
  • Test-time scaling and lexical signals: The gap between pass@1 and pass@k on hard problems exceeds 50pp, supporting multi-sample inference, while process reward models require separately trained verifiers.Budget forcing appends Wait tokens to extend reasoning, whereas the related lexical approach uses density of Wait and similar tokens to detect unproductive extended reasoning.

3 Funnel of Thoughts

Funnel of Thoughts preserves self-consistency voting while reducing computation by banking early answers and pruning active rollouts with high hesitation-marker density. Its design is motivated by early commitment predicting correctness, late generation being disproportionately expensive, and hesitation density forecasting unproductive trajectories.

  • Motivation: Correct rollouts commit after 5,575 tokens on average, versus 16,128 for incorrect rollouts; 2.1% never commit and consume the maximum budget.These findings motivate preserving committed answers before pruning.
  • Motivation: Incorrect rollouts cost 6.0 times the attention FLOPs of correct rollouts, while never-committing rollouts cost 17.4 times.Together, these rollout types comprise under a tenth of the pool.
  • Motivation: Hesitation-marker density yields a monotonic accuracy decline across deciles, with r=−0.82, rpb=−0.30 per rollout, and a 31pp D1–D10 gap.Density also forecasts final rollout length from the visible prefix, with r=+0.13 to +0.41 across models.
  • Method: At fixed token-count checkpoints, FoT moves answered rollouts into a vote bank, retains low-density active rollouts, and returns plurality across both groups.The method always retains at least 2 active rollouts, and its only hyperparameters are the checkpoint schedule and fixed keep ratio ρ.
  • Method: FoT measures hesitation density cumulatively over generated text and ranks active rollouts in ascending density, retaining ⌊|A| · ρ⌋ survivors.Hesitation markers include “Wait,”, “actually”, “perhaps”, and 18 others.

4 Experiments

Across six models and four math benchmarks, FoT preserves SC@32 accuracy while substantially reducing attention FLOPs, outperforming or complementing efficient self-consistency baselines on difficult samples. It also transfers across architectures and domains, reduces online latency, and benefits from large rollout pools and hesitation-density pruning.

  • Main results: FoT@32 matches SC@32 on 3,580 of 3,600 paired instances, with 12 wins and 8 losses, while roughly halving attention FLOPs.The difference is not significant: McNemar’s exact test gives p=0.50, with a paired-bootstrap 95% CI of [−0.14, +0.36]pp.
  • Baseline comparisons: FoT is more robust than Adaptive Consistency on hard samples, where consensus is slow but individual trajectories still expose pruning signals.Adaptive Consistency is strongest on easier cells with fast consensus, whereas FoT’s savings grow as consensus slows; the methods can compose because their decision axes are independent.
  • Baseline comparisons: 67.1% FLOP savings from Slim-SC on AIME25 coincide with an accuracy drop from 69.4 to 61.7, showing that similarity pruning can discard useful diversity.On hard problems, independently useful correct solutions may share surface structure, making inter-trajectory similarity an unsafe redundancy signal.
  • Generalization: FoT preserves SC@32 accuracy while saving 51.9% attention FLOPs on average across four held-out models with an unchanged configuration.Its relative within-pool pruning rule avoids model-specific calibration despite differing hesitation-marker densities; the gain is interpreted as capturing Phi-4-reasoning’s heavy no-answer tail.
  • Generalization: FoT improves mean per-rollout pass rate while reducing FLOPs on out-of-domain tasks, with pass@k declining modestly by −2.8pp on average.The signal acts during reasoning before answer extraction, including boxed numbers, multiple-choice options, and executable code.
  • Online deployment: 56.1% attention-FLOP and 37.6% wall-time reductions let FoT@32 solve 24 of 30 AIME24 problems, versus 23 for SC@32.Cumulative generated-token footprint falls from 12.88M to 8.51M tokens, reducing KV-cache occupancy.

5 Discussion

FoT’s reliable signal is distributional rather than per-rollout correctness: high hesitation-density rollouts are disproportionately wrong. Its within-problem comparative pruning preserves voting diversity while removing within-trajectory waste, supporting transfer across models and robustness in hard or shifted settings.

  • Discussion: r = −0.83: Error rate rises monotonically with hesitation density, making the high-density tail disproportionately wrong despite only moderate aggregate separation.The aggregate point-biserial correlation is rpb= −0.30, so FoT targets the distributional tail rather than predicting each rollout’s correctness.
  • Discussion: FoT ranks rollouts comparatively within the same model, problem, and checkpoint instead of using a model-specific marker-frequency threshold.This ranking captures shared patterns of repeated revision, backtracking, and non-commitment despite differing absolute frequencies across LRMs.
  • Discussion: FoT preserves pool-level trajectory diversity while pruning within-trajectory waste before it dominates the compute budget.This distinguishes it from efficient-SC baselines, which are strongest when answer distributions stabilize quickly or trajectories are genuinely redundant.
  • Discussion: FoT’s advantage is most persuasive on hard cells, held-out models, and domain shifts, where robustness matters more than early consensus.In harder regimes, consensus may be slow and superficially similar derivations can still provide useful voting evidence.

6 Limitations · A Motivating Analysis

FoT is evaluated mainly on reasoning tasks with extractable final answers and depends on reliable commitment detection for early voting. Its calibration, language-specific marker kernel, checkpoint schedule, and FLOP assumptions may limit how broadly its reported efficiency applies.

  • 6 Limitations: FoT’s evaluation excludes open-ended generation, multi-turn interaction, and tasks whose correctness cannot be reduced to answer extraction.Cross-domain tests still use relatively well-defined answer or execution-based evaluation.
  • 6 Limitations: 3,000 of 3,600 model-problem pairs come from MATH500, weighting pooled statistics toward easy problems where pruning has little to remove or risk.The hard split is the more relevant reference for judging the mechanism, while a different difficulty mix would change pooled figures.
  • 6 Limitations: FoT’s early-voting accuracy depends on a commitment detector, using the \boxed{} convention in mathematics and task-specific extraction or prompting elsewhere.The pruning signal is independent of final-answer format, but the vote bank is only as reliable as the commitment detector.
  • 6 Limitations: FoT fixes its checkpoint schedule and uses an English hesitation-marker kernel, so other languages or deliberation styles may require revalidating the marker set.The relative-density rule reduces sensitivity to model-level verbosity but does not remove this domain dependence.
  • 6 Limitations: Under efficient-attention mechanisms, FoT’s FLOP savings scale down toward the token-reduction floor rather than matching the standard full-attention accounting.The reported accounting assumes standard full attention.
  • 6 Limitations: Adaptive checkpointing based on per-problem difficulty is identified as a natural extension for improving the efficiency–accuracy tradeoff.This is presented as future work rather than part of the fixed-schedule method evaluated here.
  • A Motivating Analysis: The motivating analysis ranks an operational 21-marker kernel by benchmark-weighted point-biserial correlation with correctness on the 115,200-rollout pool.The table reports rollout prevalence and the wrong-to-correct frequency ratio, with † marking markers essentially absent from correct rollouts where the ratio is unstable.

A.1 Hesitation-Marker Validation … B.5 Difficulty-Stratified Accuracy

FoT validates hesitation-marker density as an online signal of unproductive, lengthening rollouts and transfers a relative pruning rule across models. Diagnostics show that FoT preserves majority-vote behavior while helping most on difficult, low-consensus problems, with clear limits when no correct rollout exists.

  • A.1 Hesitation-Marker Validation: 21 markers were retained because their density had a significant negative benchmark-weighted correlation with rollout correctness, while positively correlated markers were excluded.The composite 21-marker signal yielded rpb = −0.30.
  • A.1 Hesitation-Marker Validation: 75.7% to 50.7% accuracy separated the lowest- and highest-density deciles, showing that the signal targets an unproductive tail rather than identifies one correct rollout.Across 115,200 rollouts, accuracy fell monotonically with density, with r= −0.82 between decile index and accuracy.
  • A.2 Marker-Selection Validation: 71% of no-answer rollouts were removed by the final checkpoint, while a shared 21-marker kernel transferred through within-pool relative ranking without retuning across models.Absolute marker density varies substantially across models, so FoT avoids a fixed cross-model density threshold.
  • A.3 Pass@1 vs. Pass@32 Performance Gap: 14.4 percentage points was the average pass@1–pass@32 gap across six models and four benchmarks, confirming substantial rollout-level stochasticity.The gap provides headroom for majority voting because correct answers often appear somewhere in the 32-rollout pool.
  • B Additional Experiments / B.1 Per-Model Accuracy Breakdown: FoT is most helpful when models retain substantial sampling variance, whereas near-ceiling models offer less room for improvement; SlimSC is less stable on harder AIME problems.These diagnostics are part of the secondary experiments comparing per-model behavior and sample-axis baselines.
  • B.2 Discordant Instance Analysis / B.3 Vote-Concentration Diagnostic: 0.2% of cases changed final answers when SC@32 had a plurality share above 50%, while FoT won more often than it lost in discordant cases.FoT losses clustered on harder problems with low pass rates, and four pairs without extractable SC answers were excluded from the vote-concentration diagnostic.
  • B.4 Difficulty Adaptation: MATH500 retained accuracy parity while saving less compute than AIME/AMC because easier problems produce shorter rollouts with less hesitation tail to prune.The savings therefore track late-stage reasoning waste rather than applying a uniform reduction across benchmarks.
  • B.5 Difficulty-Stratified Accuracy: 4.8–5.8pp above SC@32 was FoT’s accuracy advantage in low-pass@1 bins, where the pass@1–pass@32 gap is largest and votes are least concentrated.The p1=0 bin is unreachable for prune-only methods because no correct rollout exists, while p1>0.5 is saturated at ceiling.

B.6 Sample-Axis Baselines on a Unified FLOP Basis · C Additional Ablations

FoT and adaptive consistency methods operate on different axes, with DSC saving more FLOPs at equal pooled accuracy while FoT’s savings increase on harder problems. Combining the independent methods yields substantially larger full-model savings with limited accuracy costs.

  • B.6 Sample-Axis Baselines on a Unified FLOP Basis: 69.4% vs. 28.8%: DSC saves more full-model FLOPs than FoT at equal pooled accuracy against SC@32.The comparison uses math_verify grading and includes attention, FFN, and projection FLOPs.
  • B.6 Sample-Axis Baselines on a Unified FLOP Basis: 11.5–13.2%: the offline Certaindex proxy preserves SC accuracy while reducing savings relative to DSC and FoT.The proxy is indicative because it can trigger later than the original online procedure.
  • B.6 Sample-Axis Baselines on a Unified FLOP Basis: 75.0% to 54.4%: DSC’s savings fall from MATH500 to AIME24/25 as consensus slows.DSC acts on the sample axis by deciding how many rollouts to draw.
  • B.6 Sample-Axis Baselines on a Unified FLOP Basis: 21.9% to 44.0%: FoT’s savings rise from MATH500 to AIME24/25 because harder problems have longer unproductive tails.FoT prunes along the token axis rather than the sample axis.
  • B.6 Sample-Axis Baselines on a Unified FLOP Basis: 80.9% and 75.7%: FoT+DSC achieves these full-model savings on all and hard problems, respectively.The methods compose because their token-axis and sample-axis operations are independent.
  • B.6 Sample-Axis Baselines on a Unified FLOP Basis: −0.4 and −1.4pp: FoT+DSC incurs these accuracy costs on all and hard problems, respectively.The reported savings and accuracy costs are measured against SC@32.

C.1 Hyperparameter Sensitivity … C.4 Per-Model Config Transfer

The study selects a universal Mid {6K, 10K, 14K}, ρ=0.67 configuration as a conservative joint Pareto choice balancing math performance with out-of-distribution gains. Sensitivity analyses show that pruning is most reliable with later checkpoints and larger rollout pools, while per-model tuning provides limited, non-transferable benefits.

  • C.1 Hyperparameter Sensitivity: +15 problems on GPQA-Diamond (+1.26pp) and +14 problems across four math benchmarks (+2.33pp) make the joint winner preferable to the math-only winner despite 9 fewer math problems.The joint configuration saves 48.7% FLOPs, versus 47.9% for the math-only configuration.
  • C.1 Hyperparameter Sensitivity: At k≥24, the default schedule’s accuracy gap becomes slightly positive, supporting FoT primarily as a replacement for large-pool SC rather than a small-k method.With small initial pools, pruning is too aggressive and the gap is unfavorable.
  • C.2 Checkpoint and Threshold Sensitivity: Early checkpoints consistently reduce accuracy, reaching ∆≤−2 and as low as −15 at ρ=0.50, because hesitation signals have not accumulated sufficiently.Mid, Math-only, and Late schedules remain viable at ρ ∈{0.67, 0.75}, with ∆≥0.
  • C.3 Per-Model Calibration: Per-model calibration improves the aggregate from ∆=+4 with the universal configuration to ∆=+17 on 3,600 model-problem pairs, a marginal gain of 13 problems.DS-R1-1.5B contributes ∆=+10 versus universal ∆=+3, while the remaining five models gain at most 2 problems each.
  • C.3 Per-Model Calibration: Three of six models select the Math-only schedule, but preferred schedules are unstable enough that the universal configuration remains the default.The universal configuration is Mid {6K, 10K, 14K}, ρ=0.67.
  • C.4 Per-Model Config Transfer: ∆=+9 on 3,600 pairs is the highest cross-model math aggregate for the Math-only {6K, 11K, 16K}, ρ=0.67 configuration, versus ∆=+4 for the universal Mid configuration.The universal Mid configuration is retained as the joint Pareto choice balancing math performance against out-of-distribution generalization.
  • C.4 Per-Model Config Transfer: No per-model-tuned configuration dominates the universal configuration across math and out-of-distribution evaluations simultaneously.This transfer result supports using one universal configuration rather than model-specific settings.

C.5 Kernel-Size Robustness

Funnel of Thoughts is robust to the hesitation-marker kernel composition: accuracy remains essentially unchanged across kernel sizes, while FLOP savings stay nearly constant. The 21-token kernel is retained as the calibrated default, though smaller kernels down to top-3 are equivalent operating points.

  • Accuracy robustness: Accuracy varies by only −3 to +6 problems on 3,600, with all 8 settings within 0.17pp of one another.Even the top-1 kernel, perhaps alone, preserves SC@32 accuracy to within 3 problems.
  • Efficiency robustness: 50.59–50.76% FLOP saving across kernel sizes confirms pruning is driven by aggregate hesitation density rather than individual marker identity.The sweep adds markers in descending order of |rpb| while holding other hyperparameters at their paper defaults on a unified 4-benchmark pool of 3,600 model-problem pairs.
  • Kernel selection: The 21-token kernel remains the calibrated default for stability, while smaller kernels down to top-3 provide an equivalent operating point.The robustness finding indicates that the lexical signal is concentrated in a small number of canonical English hesitation markers.

C.6 Early Voting Analysis

Early voting banks committed answers before pruning and is usually neutral under mild pruning, but protects committed correct answers when aggressive pruning would otherwise delete them. At ρ=0.33, it preserves 71.1 accuracy at 80.5% attention-FLOP savings, compared with a 7.5pp loss without the safeguard.

  • Early voting mechanism: 14.7 of 32 rollouts are banked on average before pruning, and the bank contains a correct answer in 78% of instances.The rate is 71% among FoT wins and 25% among SC wins.
  • Early voting mechanism: Early voting is neutral in pooled ablations because MATH500 dominates the evaluation and mild pruning rarely endangers committed answers.MATH500 contributes 3,000 of 3,600 model-problem pairs, while the default keep ratio is ρ=0.67.
  • Early voting mechanism: 7.5pp accuracy is lost without early voting at ρ=0.33 and ∼82% attention-FLOP savings, whereas early voting holds accuracy at 71.1 with 80.5% savings.Without the safeguard, accuracy falls from 75.0 to 67.5; with it, the same aggressive setting holds 71.1.
  • Early voting mechanism: Early voting recovers committed correct answers that aggressive pruning would otherwise delete, providing a safety margin for pruning errors.Once a rollout has produced \boxed{}, its outcome is resolved rather than subject to future-spiraling prediction.

D Implementation Details … D.4 Slim-SC Baseline Configuration

The implementation evaluates FoT with reproducible rollout-generation settings, controlled online comparisons, and explicit accounting for synchronization, memory, and attention-regime effects. It also specifies comparable Adaptive Consistency and Slim-SC baseline configurations.

  • D.1 Generation Details: All rollouts use SGLang with T=0.6, top-p=0.95, top-k=30, and a 32,768-token maximum, with seeds 42 + rep_id.The rolling seed preserves reproducibility while keeping rollouts independent.
  • D.1 Generation Details: The online Pass@1, SC@32, and FoT@32 conditions share one isolated A100 GPU, fixed seed 2000, and the same rollout pool.This setup enables a controlled comparison across conditions.
  • D.1 Generation Details: FoT introduces no idle time in batch-mode SGLang because all rollouts share a generation schedule and pause at synchronized checkpoints.Continuous-batching systems such as vLLM may incur brief idle time when rollouts reach checkpoints at different times.
  • D.1 Generation Details: 12.88M to 8.51M generated tokens reduces the online KV-cache footprint by 33.9% under FoT@32 versus SC@32.Each generated token occupies one KV-cache position per layer.
  • D.2 Efficient-Attention Scaling: 48.7% full-attention savings decline toward the 23.0% token-reduction floor as the attention window narrows, while w ≈4–8K retains roughly 63–79% of savings.Accuracy and checkpoint-based KV-cache release remain unaffected by the attention regime.
  • D.3 Adaptive Consistency Baseline: Adaptive Consistency uses the default Beta stopping criterion with confidence threshold τ=0.95, stopping when posterior confidence in the majority answer exceeds τ.Sequential behavior is simulated over parallel rollouts by index-order processing and averaging across 10 random permutations per problem.
  • D.4 Slim-SC Baseline Configuration: Slim-SC reproduces the official configuration with cosine threshold τ=0.9, all-mpnet-base-v2 segment embeddings, and a 20-thought warm-up before pruning.These recommended defaults were used without threshold tuning on the evaluation data.
Loading 2608.15065v1…