PaperlayerView exact revision on arXiv

Source-linked AI summary

SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling

Yang Xiao, Chunpu Xu, Ruifeng Yuan, Jiashuo Wang, Wenjie Li, Pengfei Liu

arXiv:2512.00466v1cs.CLcs.AI

TL;DR

Uniform resource allocation can leave challenging reasoning sub-problems under-resourced while spending too much on routine steps. SCALE allocates processing modes according to sub-problem difficulty, and experiments report improved accuracy with lower token use than uniform scaling baselines.

  • Problem

    Existing test-time scaling methods allocate resources uniformly within problems despite substantial differences in the difficulty of individual sub-problems.

    Source: Introduction · p. 1

  • Method

    SCALE decomposes problems into sequential sub-problems, assesses their difficulty, assigns System 1 or System 2 processing, and executes them with context propagation.

    Source: Introduction · pp. 1–2, Mathematical Formulation · p. 3

  • Results

    Across challenging mathematical reasoning benchmarks, SCALE outperforms uniform scaling; on AIME25 with Qwen3-32B, accuracy rises from 57.50% to 71.25% while token use is 33-53% lower than baseline methods such as InftyThink.

    Source: Introduction · p. 2, Main Results · p. 5

  • Takeaways & Limitations

    The framework is presented as applicable both to improving existing reasoning models and to helping non-reasoning models acquire deep-thinking capabilities through supervised fine-tuning on SCALE-generated traces.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Test-time compute scaling has emerged as a powerful paradigm for enhancing mathematical reasoning in large language models (LLMs) by allocating additional computational resources during inference. However, current methods employ uniform resource distribution across all reasoning sub-problems, creating fundamental bottlenecks where challenging sub-problems receive insufficient attention while routine operations consume disproportionate resources. This uniform allocation creates performance bottlenecks where additional computational resources yield diminishing returns. Inspired by dual-process theory, we propose \textbf{SCALE} (Selective Resource Allocation), a framework that selectively allocates computational resources based on sub-problem difficulty. SCALE operates through four stages: (1) problem decomposition into sequential reasoning sub-problems, (2) difficulty assessment of each sub-problem to distinguish between routine operations and computationally challenging sub-problems, (3) selective processing mode assignment between System 1 for simple sub-problems and System 2 for complex ones, and (4) sequential execution with context propagation. By concentrating resources on challenging sub-problems while processing routine operations efficiently, SCALE achieves substantial performance improvements with superior resource utilization. Extensive experiments demonstrate that SCALE significantly outperforms uniform scaling baselines, achieving accuracy improvements of up to 13.75 percentage points (57.50% to 71.25% on AIME25) while reducing computational costs by 33%-53%, representing a major advance in test-time scaling that addresses fundamental limitations of current approaches.

Introduction

Uniform allocation can waste computation on routine steps while underfunding difficult sub-problems. SCALE assesses sub-problem difficulty to choose processing modes, and experiments report accuracy gains alongside lower token use.

Introduction

  • Uniform allocation can underuse resources on routine operations while giving challenging, solution-determining sub-problems insufficient attention.The paper identifies this within-problem mismatch as a bottleneck where additional budget yields diminishing returns.Source: Introduction · p. 1
  • SCALE decomposes problems into sub-problems, assesses their difficulty, selects System 1 or System 2, and executes sequentially with context propagation.Its selective allocation is motivated by dual-process theory.Source: Introduction · pp. 1–2, Figure 1 · p. 2
  • Up to 13.75 percentage points of accuracy improvement (57.50% to 71.25% on AIME25 with Qwen3-32B) accompanies 33-53% lower token use than baseline methods such as InftyThink.Experiments across challenging mathematical reasoning benchmarks report improvements over uniform inference-time scaling baselines.Source: Introduction · p. 2
  • The contributions span improving existing reasoning models and enabling non-reasoning models to acquire deep-thinking capabilities through supervised fine-tuning on SCALE-generated traces.The paper reports substantial performance improvements in mathematical reasoning tasks.Source: Introduction · p. 2

Method

SCALE allocates processing modes according to sub-problem difficulty within a sequential, context-aware reasoning pipeline. Its method includes selecting a decomposition, assessing difficulty, assigning modes, and integrating sub-problem solutions into the final answer.

Method

Problem Decomposition

  • The decomposition stage produces ordered sub-problems with defined objectives and dependencies; SCALE generates alternatives and selects one using quality criteria.The criteria include logical correctness, clarity, completeness, and relevance to the final solution.Source: Problem Decomposition · p. 3

Difficulty Assessment

  • For each sub-problem, difficulty assessment considers computational complexity, mathematical sophistication, reasoning depth, and solution uncertainty.Assessment uses accumulated context containing the original problem and previously solved sub-problems.Source: Difficulty Assessment · p. 3

Adaptive Mode Selection

  • SCALE selects processing mode by difficulty: System 1 handles straightforward sub-problems, while System 2 handles complex ones requiring extensive reasoning.Modern models with mode-switching capability provide the basis for this adaptive selection.Source: Adaptive Mode Selection · p. 3

Sequential Execution with Context Propagation

Experimental Setup

The experiments test SCALE both as an inference-time prompt framework for reasoning models and as a source of traces for fine-tuning non-reasoning models. Both settings use mathematical reasoning benchmarks and assess accuracy and resource use.

Experimental settings

  • SCALE is evaluated in two settings: prompt-based inference-time improvement of reasoning models and supervised fine-tuning of non-reasoning models with SCALE-generated traces.The first requires no additional training; the second uses synthetic reasoning traces as training data.Source: Experimental Setup · p. 4, Setting 2: Enhancing Non-Reasoning… · p. 4

Reasoning-model evaluation

  • The reasoning-model setting compares SCALE across four reasoning-capable models against CoT, InftyThink, and Majority Voting.The models are Qwen3-32B, QwQ, Distill-Llama-70B, and Distill-Qwen-32B.Source: Setting 1: Enhancing Reasoning Model… · p. 4

Trace generation and fine-tuning

  • For the fine-tuning setting, SCALE-generated traces from QwQ are filtered against LIMOPro answers to create 800 question-response pairs.The models are fine-tuned using the same training configurations as LIMOPro.Source: Setting 2: Enhancing Non-Reasoning… · p. 4

Benchmarks and metrics

  • The evaluation covers AIME24, AIME25, and AMC23, spanning competition-level mathematics, recent unseen challenges, and a broader range of topics.Both experimental settings use these three mathematical reasoning datasets.Source: Evaluation Benchmarks and Metrics · p. 4, Experimental Setup · p. 4
  • Effectiveness and efficiency are measured with pass@1 accuracy (Acc), tokens per iteration or sub-problem (Tpi), and total response tokens per problem (Tok).Acc is obtained by sampling each model eight times; Tok is used as a measure correlated with computational cost and inference time.Source: Evaluation Benchmarks and Metrics · p. 4

Enhancing Reasoning Model Performance

SCALE improves reasoning-model accuracy over uniform-scaling baselines while using fewer resources. Results also show how performance and token use vary with the System 2 budget and difficulty threshold.

Overcoming Performance Bottlenecks

  • 13.75 percentage points: Qwen3-32B-SCALE improves AIME25 accuracy from 57.50% to 71.25% over CoT.Across Qwen3-32B, AIME24 accuracy rises from 73.33% to 82.92%; Table 1 compares SCALE with baseline methods across benchmarks.Source: Main Results · p. 4, Table 1 · p. 6

Enhanced Resource Utilization Through Selective Allocation

  • 33–53% lower total token usage than InftyThink accompanies higher accuracy across model-dataset combinations.For Qwen3-32B on AIME25, SCALE uses 26,643 tokens versus InftyThink’s 36,640, with accuracy of 71.25% versus 70.00%.Source: Main Results · p. 5

Model-Agnostic Bottleneck Resolution

  • SCALE’s average accuracy improvement over CoT is 3.41 percentage points across models and datasets, with particular effectiveness on AIME benchmarks.The reported consistency spans Qwen3-32B, QwQ, and Llama variants.Source: Main Results · p. 5

Inference-Time Scaling Properties of SCALE

  • AIME24 and AIME25 show steep, almost linear scaling, while AMC23 improves more gradually and begins to plateau as the System 2 token limit rises.The authors attribute the difference to AIME’s higher proportion of difficult sub-problems and AMC23’s relatively greater share of routine computations; the experiment varies the System 2 limit from 4,096 to 32,768 tokens.Source: Inference-Time Scaling Properties of… · p. 5, Figure 2 · p. 5

Difficulty Threshold Analysis

  • At τ = 0.2, accuracy reaches 78.75% on AIME24 and 63.75% on AIME25, with approximately 75% of sub-problems assigned to System 2.Accuracy decreases as the threshold increases; Table 2 analyzes threshold values for Qwen3-32B-SCALE.Source: Difficulty Threshold Analysis · p. 6, Table 2 · p. 6
  • Raising τ from 0.2 to 0.9 reduces total tokens by 63% on AIME24 and 71% on AIME25, while the share of hard sub-problems falls to 0%.The passage describes threshold choice as a trade-off between accuracy and resource constraints.Source: Difficulty Threshold Analysis · p. 6

Performance

Fine-tuning on SCALE-generated traces improves non-reasoning models across model sizes and benchmarks, with especially large reported gains on AIME problems. The results also describe cross-architecture generalization and transfer of reasoning patterns.

Significant Performance Gains Across All Benchmarks

  • 38.93 percentage points: fine-tuning Llama3.3-70B-Instruct with SCALE-generated traces raises AIME24 accuracy from 24.58% to 63.51%.Across benchmarks, reported gains for fine-tuned models range from 4.69 to 39.41 percentage points.Source: Performance · p. 7

Cross-Architecture Generalization and Scaling Effects

  • SCALE’s gains generalize across architectures and sizes, while the Qwen2.5 32B and 72B models have similar AIME24 accuracy: 53.33% and 54.17%.The authors report larger gains for the 32B and 72B Qwen2.5 models than for the 14B model on challenging AIME problems.Source: Performance · p. 7

Enhanced Reasoning Pattern Transfer

  • On AIME benchmarks, larger fine-tuned models reach 44–63% accuracy after starting below 25%, which the authors describe as transfer of sophisticated reasoning patterns.This transfer is reported for SCALE-generated traces used in fine-tuning.Source: Performance · p. 7

Related Work

Prior reasoning approaches span chain-of-thought prompting, process reward models, extended test-time reasoning, and adaptive allocation strategies. The paper positions SCALE as fine-grained resource allocation within sub-problems.

Related Work

  • The described progression runs from chain-of-thought prompting and process reward models to extended reasoning traces in models such as o1 and DeepSeek-R1.The passage also notes the computational overhead associated with longer reasoning sequences.Source: Related Work · p. 7
  • Adaptive methods cited in the paper adjust reasoning mode, reasoning length, or engagement, but are described as operating primarily at the problem level.Examples include mode switching, reasoning-length optimization, learning-based engagement, and hybrid approaches.Source: Related Work · p. 7
  • SCALE is presented as addressing that gap through fine-grained resource allocation at the sub-problem level.The passage characterizes this approach as selective cognitive resource distribution.Source: Related Work · p. 7

Conclusion

SCALE addresses inefficient uniform resource allocation in mathematical test-time compute scaling by assigning processing modes according to sub-problem difficulty. Experiments show accuracy improvements and superior resource-utilization efficiency compared with uniform scaling approaches.

Conclusion

  • SCALE addresses the bottleneck of inefficient uniform resource allocation in mathematical test-time compute scaling.The conclusion describes this allocation as preventing effective scaling despite increased computational budgets.Source: Conclusion · p. 7
  • The framework decomposes problems into sequential sub-problems, assesses their difficulty, and dynamically assigns appropriate processing modes.It is inspired by dual-process theory and concentrates resources where they can maximize impact.Source: Conclusion · p. 7
  • Experiments demonstrate substantial accuracy improvements and superior resource-utilization efficiency compared with uniform scaling approaches.Source: Conclusion · p. 7

Related papers

Loading 2512.00466v1…