PaperlayerView exact revision on arXiv

Source-linked AI summary

s1: Simple test-time scaling

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, Tatsunori Hashimoto

arXiv:2501.19393v3cs.CLcs.AIcs.LG

TL;DR

The paper asks whether strong reasoning and test-time scaling can be achieved with a simple, transparent recipe. It curates a 1,000-example reasoning dataset, applies supervised fine-tuning and budget forcing, and reports competitive reasoning performance with scaling as test-time compute increases.

  • Problem

    OpenAI o1 demonstrated strong gains from scaling test-time compute, but its methodology was not publicly shared, motivating simpler open replication approaches.

  • Method

    The authors curate s1K, a 1,000-example reasoning dataset, fine-tune an off-the-shelf model with next-token prediction, and control thinking duration using budget forcing.

  • Results

    Supervised fine-tuning on 1,000 examples produces a competitive reasoning model, while budget forcing enables performance scaling with more test-time compute.

  • Takeaways & Limitations

    Careful selection using difficulty, diversity, and quality is important, since alternative selection strategies perform around −30% worse on AIME24 on average.

  • Takeaways & Limitations

    The paper leaves open whether reinforcement learning could improve extrapolation or enable test-time scaling methods beyond budget forcing.

Abstract

from arXiv · show

Test-time scaling is a promising new approach to language modeling that uses extra test-time compute to improve performance. Recently, OpenAI's o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. We seek the simplest approach to achieve test-time scaling and strong reasoning performance. First, we curate a small dataset s1K of 1,000 questions paired with reasoning traces relying on three criteria we validate through ablations: difficulty, diversity, and quality. Second, we develop budget forcing to control test-time compute by forcefully terminating the model's thinking process or lengthening it by appending "Wait" multiple times to the model's generation when it tries to end. This can lead the model to double-check its answer, often fixing incorrect reasoning steps. After supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and equipping it with budget forcing, our model s1-32B exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). Further, scaling s1-32B with budget forcing allows extrapolating beyond its performance without test-time intervention: from 50% to 57% on AIME24. Our model, data, and code are open-source at https://github.com/simplescaling/s1

1. Introduction

The paper asks whether simple, open methods can reproduce both strong reasoning and test-time scaling, then proposes a 1,000-example SFT recipe combined with budget forcing. The resulting approach supports scaling with additional test-time compute and competitive performance against o1-preview.

  • Motivation: Test-time scaling increases compute during inference to improve language-model results, but clear open replications of o1's scaling behavior were lacking.The paper frames this as a gap after o1 demonstrated strong reasoning and gains from scaling test-time compute.
  • Approach: Training on only 1,000 samples with next-token prediction and budget forcing produces a reasoning model whose performance scales with more test-time compute.The method uses supervised fine-tuning on s1K and a decoding-time intervention to control thinking duration.
  • Approach: Budget forcing ends thinking after a maximum token budget or encourages further reasoning by suppressing the delimiter and appending “Wait”.These two controls respectively shorten generation or lengthen it to encourage exploration and reflection.
  • Evidence: Careful selection using difficulty, diversity, and quality is important: isolated selection strategies perform around −30% worse on AIME24 on average.Training on the full 59K-example pool also provides no substantial gain over the selected 1K subset.
  • Outcome: The resulting s1-32B is competitive with o1-preview, while the model, dataset, and code are released openly.The paper presents open release as part of its contribution alongside the model and ablations.

2. Reasoning data curation to create s1K

The authors build s1K by filtering a 59K-question pool and selecting 1,000 examples according to quality, difficulty, and diversity. The resulting set spans multiple reasoning domains and pairs questions with generated reasoning traces and solutions.

  • Pool construction: The initial pool contains 59,029 questions from 16 sources guided by quality, difficulty, and diversity.Sources include existing mathematical and cross-disciplinary datasets plus original quantitative-reasoning datasets.
  • Pool construction: Each question receives a Gemini-generated reasoning trace and solution, producing 59K question-trace-solution triplets that are decontaminated and deduplicated.Decontamination targets MATH500, GPQA Diamond, and AIME24 evaluation questions.
  • Filtering: The pool is filtered to a minimal 1,000-sample set because the authors seek a simple approach using minimal resources.Filtering preserves the three guiding principles rather than training directly on all 59K questions.
  • Filtering: Difficulty is estimated from model correctness and reasoning-trace length, using Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct with Claude 3.5 Sonnet grading.Longer reasoning traces are used as one indicator of problem difficulty.
  • Filtering: Diversity is organized by domain classification, then selection samples across 50 domains while favoring longer reasoning traces.Domains are classified with Claude 3.5 Sonnet using an MSC-based taxonomy that also includes sciences beyond mathematics.
  • Quality boundary: The final s1K traces are not uniformly correct: the grader marks 53.6% correct in s1K and 63.0% in s1K-1.1.The authors retain some incorrect distilled generations because they aim to capture reasoning processes, not only correct solutions.

3. Test-time scaling

The paper distinguishes sequential from parallel test-time scaling and focuses on sequential computation that can build on intermediate reasoning. It introduces budget forcing to control thinking length and evaluates scaling through controllability, slope, and maximum performance.

  • Scaling framework: Sequential scaling lets later computations depend on earlier reasoning, whereas parallel scaling runs independent computations such as majority voting.The paper focuses on sequential scaling because later computation can support deeper reasoning and iterative refinement.
  • Budget forcing: Budget forcing controls thinking duration by forcing a maximum and/or minimum number of thinking tokens.Maximum budgets append the end-of-thinking delimiter, while minimum budgets suppress it and can append “Wait”.
  • Evaluation: The evaluation measures controllability, scaling slope, and performance while varying test-time compute on a fixed benchmark.The resulting piece-wise linear function uses thinking tokens on the x-axis and accuracy on the y-axis.
  • Evaluation: Controllability measures how closely generated thinking tokens follow the prescribed compute bounds, with 100% representing perfect control.The metric is reported as a percentage and usually constrains the maximum compute.
  • Evaluation: Performance is the maximum benchmark accuracy, while useful scaling requires a positive average slope; excessive scaling can flatten or fail under control and context limits.The paper treats monotonic improvement and eventual limitations as distinct properties of scaling methods.

4. Results

The results evaluate s1-32B against reasoning models and benchmarks, showing strong sample efficiency and gains from sequential test-time scaling with budget forcing.

  • Training and evaluation: 26 minutes on 16 NVIDIA H100 GPUs produced s1-32B through supervised finetuning of Qwen2.5-32B-Instruct.
  • Training and evaluation: s1-32B is evaluated on AIME24, MATH500, and GPQA Diamond using accuracy, typically with greedy decoding.
  • Model comparisons: The comparison includes OpenAI o1, DeepSeek r1, QwQ-32B-preview, Sky-T1-32B-Preview, Bespoke-32B, and Gemini 2.0 Flash Thinking Experimental.
  • Test-time scaling: Sequential budget forcing outperforms parallel majority voting on the Qwen2.5-32B-Instruct base model as test-time compute increases.
  • Sample efficiency: s1-32B is the most sample-efficient open data reasoning model and nearly matches Gemini 2.0 Thinking on AIME24 after training on 1,000 samples.

5. Ablations

The ablations show that carefully combining data-selection criteria and choosing suitable test-time scaling methods are central to s1-32B’s performance. Budget forcing offers strong control and scaling, while rejection sampling can exhibit inverse scaling as longer generations become more error-prone.

  • 5.1. Data Quantity, Diversity, and Difficulty: Combining quality, difficulty, and diversity in s1K is key for sample-efficient reasoning training.Training on all 59K examples used 394 H100 GPU hours, compared with 7 H100 GPU hours for s1-32B.
  • 5.2. Test-time scaling methods: Budget forcing provides perfect control, good scaling, and the best AIME24 score among the compared test-time scaling methods.The authors therefore use budget forcing for s1-32B’s reported scaling results.
  • 5.2. Test-time scaling methods: Suppressing the end-of-thinking delimiter too often can produce repetitive loops rather than continued reasoning.Performance on AIME24 eventually flattens after six times more test-time compute.
  • 5.2. Test-time scaling methods: Longer rejection-sampled generations can be wrong because they often reflect backtracking or self-questioning after an initial mistake.A question was answered correctly under a ≤4000-token setting but not under ≤8000 tokens.

6. Discussion and related work

The paper positions sample-efficient supervised fine-tuning and budget forcing as simple routes to competitive reasoning and test-time scaling. It also identifies flattening, context-window limits, and evaluation-compute trade-offs, while exploring parallel methods for scaling beyond sequential limits.

  • Sample-efficient reasoning: SFT on only 1,000 examples produces a competitive reasoning model matching o1-preview and lying on the pareto frontier.The authors hypothesize that pretraining already provides reasoning ability, while sample-efficient fine-tuning activates it.
  • Test-time scaling: Budget forcing reproduces OpenAI’s test-time scaling curves by controlling the model’s thinking duration during decoding.The method can terminate thinking early or extend it by suppressing the end delimiter and appending “Wait.”
  • Test-time scaling: Budget forcing improves AIME24 performance from 50% to 57%, but further scaling eventually flattens and is constrained by the model’s context window.Scaling down test-time compute behaves predictably and avoids these constraints across a wide range of accuracies.
  • Future directions: Further test-time extrapolation remains an open direction, including varied forcing strings, penalties or temperature changes, and reinforcement-learning-based reasoning models.The paper proposes Control, Scaling, and Performance as metrics for future progress.
  • Parallel scaling: Parallel scaling complements sequential scaling, with REBASE scaling better than majority voting in the reported Figure 7 comparison.REBASE adds reward-model computation, while prompting for 512 sequential steps causes 12 of 30 questions to exceed the context window.

Impact Statement

The paper argues that transparent reasoning models could improve productivity and support scientific breakthroughs. It aims to advance reasoning research openly because recent systems such as o1 and r1 lack transparency.

  • Potential impact: Open reasoning models could assist complex decision-making and drive scientific breakthroughs.The statement frames these benefits as potential applications of strong reasoning capabilities.
  • Open research: The work seeks to foster open innovation and collaboration by addressing the limited transparency of recent reasoning advances.The cited passage specifically names OpenAI’s o1 and DeepSeek’s r1.

B. Evaluation determinism

The evaluation pipeline uses vLLM but finds that scores can vary across runs even with identical seeds and greedy sampling. Long reasoning traces can amplify small numerical differences into different final answers.

  • Sources of nondeterminism: Evaluation scores can change significantly across runs despite using the same random seeds and greedy sampling.The authors attribute variation to implementation factors including batch size, continuing generations, and tensor parallelism.
  • Sources of nondeterminism: Long reasoning traces can cause small numeric changes to snowball into entirely different answers.Generations may match for thousands of tokens before diverging at one token and ending with different answers.
  • Measurement: The dataset summary reports token counts measured with the Qwen-2.5 tokenizer.This measurement convention is stated in the caption for Table 6.

C.2. Dataset composition for full 59K questions

The full 59K-question pool is documented through its composition, selection pipeline, grading procedure, and benchmark-overlap filtering. The pipeline constructs s1K through staged sampling from candidate questions and domains.

  • Dataset composition: The full dataset contains 59K questions, with thinking and response lengths measured using the Qwen2.5-32B-Instruct tokenizer.AIME24 and the 2022–2023 AIME questions used during development are excluded from the relevant pool.
  • Grading: Correctness grading uses a Figure 8 prompt, with Claude 3.5 used generally and Claude 3.7 used for the final 1,000 samples.The grading process supports dataset selection in Section 2.
  • Selection pipeline: The selection algorithm combines benchmark-specific high-quality samples with a two-stage diversity-selection procedure.Algorithm 1 is described as selecting 1,000 questions from an input set of 24,496 questions with features.
  • Evaluation separation: Selected examples are filtered by excluding questions with more than an 8-gram overlap with MATH500, GPQA Diamond, or AIME24.The filtering is intended to prevent overlap with evaluation benchmarks.

D. Training details

The model is further fine-tuned from Qwen2.5-32B-Instruct for reasoning, using delimiters to separate thinking from answering; training dynamics are reported for s1-32B.

  • Model and format: Qwen2.5-32B-Instruct is further fine-tuned for reasoning rather than trained from scratch.The implementation uses an already pretrained and instruction-tuned model.
  • Model and format: Token delimiters separate the model’s thinking stage from its answering stage.The thinking stage is enclosed with special instruction delimiters.
  • Training dynamics: Training dynamics are reported for s1-32B.Figure 9 presents the training dynamics of the model on s1K.

D.1. Training Ablations: Sequence length

The training sequence-length ablation shows a trade-off between training truncation and test-time reasoning length, with shorter training sequences producing longer reasoning traces.

  • Evaluation: Table 8 reports accuracy and average thinking tokens per sample for the training sequence-length ablation.The table caption states that higher accuracy and fewer thinking tokens are preferred.
  • Sequence length: Shorter training sequence lengths lead to longer reasoning traces at test time.The paper attributes this to more answer sections being cut off during training.
  • Sequence length: Longer training sequences expose more complete samples, including the section where the model answers.This changes how often the answer section appears intact during training.

D.2. Training Samples

The training-sample appendix presents s1K examples spanning probability, mathematical games, coin-flip expectations, and physics-style quantitative reasoning, including questions and generated solutions or traces.

  • Probability samples: s1K includes a probability sample framed as a PhD qualifying-exam problem with a proof-oriented solution.Table 9 identifies the sample as stemming from s1-prob.
  • Probability samples: One sample asks for the expected number of Alice’s coin flips conditioned on Bob flipping more times.The accompanying reasoning models both flip counts as independent geometric random variables.
  • Probability samples: The coin-flip solution uses symmetry and first-flip cases to derive the conditional expectation.The case analysis distinguishes HT, TH, and TT outcomes under the condition that Bob flips more often.
  • Quantitative samples: The appendix also contains a physics calculation estimating a surface temperature from power density and black-body radiation.The worked estimate is approximately 49,800 K, closest to the 50,000 K answer choice.

E.1. Sequential scaling ablations

The paper evaluates token-, step-, and class-conditional controls for sequential test-time scaling, finding that coarse step control can scale thinking time but requires intervention for reliable adherence.

  • Evaluation: The appendix reports token-, step-, and class-conditional scaling results on AIME24.Tables 12–14 summarize these controls, with class-conditional results reporting accuracy and average thinking tokens per sample.
  • Token-conditional control: Token-conditional control asks the model to generate a specified number of reasoning tokens.The method buckets training-trace lengths into powers of two and adds corresponding instructions.
  • Step-conditional control: Step-conditional control partitions reasoning traces into steps and instructs the model to think for a target number of steps.Steps are split on double newlines and bucketed into powers of two.
  • Step-conditional control: The model struggles to adhere to step limits and may continue counting into negative steps.Automatically stopping at zero and appending the answer delimiter produces perfect step adherence, although other problems remain.
  • Step-conditional control: More allowed steps correlate with more total thinking tokens and better performance.The model partly compensates for fewer steps by making each step longer, but larger step budgets still increase total thinking tokens.
  • Class-conditional control: Class-conditional control exposes three reasoning-effort settings: low, medium, and high.The discussion relates these settings to controlling response speed and reasoning-token usage.

E.2. Examples for rejection sampling ablation

The section presents rejection-sampling examples for s1-32B on AIME24, contrasting an incorrect generation with a correct generation and recording the sampling effort required to meet thinking-token limits.

  • The examples generate reasoning traces and responses from the question alone at temperature 1.The procedure generates until the reasoning trace satisfies the specified token threshold.
  • Rejection sampling required an average of 199 tries per example to obtain thinking traces of at most 4,000 or 8,000 tokens.The table describes separate thresholds of ≤4000 and ≤8000 tokens for the two examples.
Loading 2501.19393v3…