PaperlayerView exact revision on arXiv

Source-linked AI summary

RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?

Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, Zhijiang Guo, Yaodong Yang, Muhan Zhang, Debing Zhang

arXiv:2501.11284v1cs.LGcs.AIcs.CL

TL;DR

The paper asks whether scaling Long-COT data can improve slow-thinking reasoning despite limited evidence about dataset scale, model capacity, training strategies, and multimodal integration. It studies Long-COT scaling across model sizes and domains, adds reinforcement learning, and extends the approach to vision-language models. The results show scale-dependent reasoning gains from minimal data, preserved general-task performance, further RL improvements, and notable multimodal benefits.

  • Problem

    Long-COT dataset scalability and the roles of data synthesis, model capacity, training strategy, and multimodal integration remain insufficiently investigated.

    Source: §1 · p. 2

  • Method

    The paper scales and evaluates Long-COT training across model sizes and domains, adds reinforcement-learning training, and applies Long-COT to vision-language models.

    Source: §1 · pp. 2–3, §4 · p. 7, §6 · p. 11

  • Results

    Long-COT produces significant scale-dependent reasoning gains from minimal samples, preserves general-task performance, gains further from RL training, and improves multimodal task performance.

    Source: §1 · pp. 2–3, §8 · p. 13

  • Takeaways & Limitations

    Long-COT tuning can effectively trigger slow-thinking reasoning with limited data while supporting broader reasoning, reinforcement-learning, and multimodal applications.

    Source: §1 · p. 3, §8 · p. 13

    Future work still needs more scalable automated data generation using instruction-tuned models and evaluation on additional complex benchmarks.

    Source: §8 · p. 13

Abstract

from arXiv · show

Can scaling transform reasoning? In this work, we explore the untapped potential of scaling Long Chain-of-Thought (Long-CoT) data to 1000k samples, pioneering the development of a slow-thinking model, RedStar. Through extensive experiments with various LLMs and different sizes, we uncover the ingredients for specialization and scale for Long-CoT training. Surprisingly, even smaller models show significant performance gains with limited data, revealing the sample efficiency of Long-CoT and the critical role of sample difficulty in the learning process. Our findings demonstrate that Long-CoT reasoning can be effectively triggered with just a few thousand examples, while larger models achieve unparalleled improvements. We also introduce reinforcement learning (RL)-scale training as a promising direction for advancing slow-thinking systems. RedStar shines across domains: on the MATH-Hard benchmark, RedStar-code-math boosts performance from 66.2\% to 81.6\%, and on the USA Math Olympiad (AIME), it solves 46.7\% of problems using only 21k mixed-code-math datasets. In multimodal tasks like GeoQA and MathVista-GEO, RedStar-Geo achieves competitive results with minimal Long-CoT data, outperforming other slow-thinking systems like QvQ-Preview. Compared to QwQ, RedStar strikes the perfect balance between reasoning and generalizability. Our work highlights that, with careful tuning, scaling Long-CoT can unlock extraordinary reasoning capabilities-even with limited dataset and set a new standard for slow-thinking models across diverse challenges. Our data and models are released at https://huggingface.co/RedStar-Reasoning.

1 Introduction

RedStar investigates whether scaling Long-COT data improves slow-thinking reasoning across model sizes, domains, training strategies, and modalities. The results show strong gains from limited and larger datasets, with benefits extending to general, reinforcement-learning, and multimodal settings.

Data Scaling

  • Long-COT fine-tuning yields substantial improvements across a range of reasoning tasks, including with a curated set of 1.3k prompts and 4k prompt-response pairs.The study examines whether scaling Long-COT data produces a scaling effect and finds meaningful gains even from small, moderately difficult datasets.Source: §1 · p. 2
  • Larger datasets benefit smaller models, while larger models improve notably with smaller datasets, highlighting the joint importance of data scale and model capacity.The 7B-Instruct model benefits substantially from more data, whereas the 14B-Instruct model improves even with smaller datasets.Source: §1 · p. 2

Reinforcement Learning

  • Reinforcement-learning strategies further improve Long-COT reasoning quality and performance through feedback-based refinement.Both offline DPO and online PPO are reported as effective for enhancing model performance.Source: §1 · p. 3

RedStar and Multimodal Extension

  • A 4k-scale Long-COT instruction set on Qwen-32B-Instruct produces stronger reasoning consistency and complex-task handling than smaller models, while also improving multimodal geometric reasoning.The study extends step-by-step Long-COT reasoning to multimodal large language models and reports substantial gains on geometric question-answering tasks.Source: §1 · p. 3

Conclusion

  • The final RedStar model is presented as broadly effective across mathematical and multimodal reasoning challenges, demonstrating the wide impact of Long-COT tuning.The conclusion summarizes gains from limited data, scale-dependent improvements, RL-based enhancement, and vision-language applications.Source: §1 · p. 3

Generalization

  • Long-COT improves reasoning while preserving or improving performance on other domains, languages, foundational tasks, and alignment evaluations.The reported transfer suggests that Long-COT is not limited to mathematical reasoning and exhibits generalization and robustness beyond the training tasks.Source: §1 · p. 3

2 Long-COT Data Curation

The Long-COT dataset is assembled from mathematical, code, and geometric sources, then filtered by difficulty and response quality. The resulting collection combines a 1000k-sample corpus with smaller high-quality subsets for supervised fine-tuning and subsequent reinforcement learning.

Curation Pipeline

  • The curation pipeline collects prompts from diverse mathematical, code-based, and multimodal sources before difficulty filtering and response-quality evaluation.Figure 1 summarizes collection, difficulty annotation and filtering, and construction of positive and negative samples.Source: §2 · pp. 3, 5, Figure 1 · p. 4

Project Scope

  • The broader project comparison emphasizes open-source data and code while examining Long-COT across a wider range of data domains than prior slow-thinking approaches.The comparison is presented in Figure 2 as an overview of the exploration’s scope.Source: Figure 2 · p. 4

Prompt Collection

  • The source pool includes approximately 900k math problems, 25,443 TACO code problems, and 3,499 GeoQA geometric prompts with associated solutions, tests, or images.Math data combines MetaMathQA, Numica-CoT, and NuminaMath-CoT sources; code data includes fine-grained labels and test cases.Source: §2 · pp. 3, 5

Difficulty Scoring and Augmentation

  • A difficulty model selects problems at levels 7 and above for augmentation, with more QwQ samples generated for harder prompts and correctness checked against official answers.The procedure targets higher-difficulty problems while retaining answer-based validation.Source: §2 · p. 5

Response Verification

  • Repeated sampling and rule-based verification retain correct math and executable code responses as positive SFT examples, while failed responses become negative samples for later RL training.For code, responses are tested against TACO test cases, yielding 16,713 retained samples.Source: §2 · p. 5

Final Dataset

  • The final dataset contains 1000k samples, including 220k prompts, plus a high-quality subset of 1.3k prompts and 4k correct prompt-response pairs at difficulty levels 7–9.These subsets are used for supervised fine-tuning and improve models based on Qwen2-Math-7B and Qwen2.5-32B-Instruct.Source: §2 · p. 5

3 Scaling Data and Models for Long-COT Efficiency

Scaling Long-COT data improves mathematical reasoning, with gains shaped by model capacity, specialization, and prompt difficulty. Larger or specialized models use small datasets more efficiently, while weaker models benefit from greater data volume.

3.1 Long-COT Data Scaling Performance and Efficiency Analysis

  • 1000k Long-COT examples raise 7B-Math-Instruct performance from 36.8 to 45.4, while 14B-Instruct reaches an average score of 44.3 across the same configurations.The supplied results describe continuous improvement as Long-COT data increases, especially on difficult reasoning tasks.Source: §3.1 · p. 6
  • 8.5 and 4.2 points are gained from 0 to 4k examples by 14B-Instruct and 7B-Math-Instruct, whereas 7B-Instruct shows minimal improvement.Overall gains from 0 to 1000k are around 10.0 for all models, but early-data efficiency differs substantially.Source: §3.1 · p. 6
  • Parameter scaling and math pretraining improve sample efficiency: 7B-Math-Instruct and 14B-Instruct perform strongly with small datasets, while 7B-Instruct needs 1000k data to approach them.The comparison links long-context or math pretraining and parameter scaling with more efficient Long-COT training.Source: §3.1 · p. 7
  • Long-COT data scaling improves challenging benchmarks such as Olympiad-bench and AIME24, but dataset size should be tailored to model capacity.The section emphasizes both data volume and high-quality data, while noting different sample-utilization efficiencies across models.Source: §3.1 · pp. 5, 7

3.2 Difficult Open-ended Questions Are Good Meta-data

  • Medium- and high-difficulty mathematical prompts are tested against low-difficulty prompts to measure how problem difficulty affects Long-COT synthesis.The ablation groups problems into level 3, levels 4–6, and levels 7–9, using 4k samples per difficulty level.Source: Table 2 · p. 6, §3.2 · p. 7

3.3 Model Specialization or Parameter Scaling Improves Performance

  • 45.4 is achieved by both 14B-Instruct and 7B-Math-Instruct at 1000k data, while 7B-Instruct reaches 41.7, showing benefits from capacity and specialization.Math specialization also yields 73.3 on Math-hard versus 71.3 for 7B-Instruct and 73.4 for 14B-Instruct.Source: §3.3 · p. 7
  • Larger or domain-specialized models produce more robust gains across benchmarks, particularly on High_school_league and AIME24.The supplied findings attribute stronger performance to either increased parameter count or specialized training.Source: §3.3 · p. 7

4 Reinforcement Learning Brings Further Improvement

The paper evaluates reinforcement-learning methods for improving Long-COT reasoning using rule-based rewards and preference or policy optimization. DPO and PPO achieve the strongest reported average performance, while REINFORCE++ is more variable across tasks.

  • RL training uses a rule-based reward model to filter positive and negative responses, with the RL dataset drawn from a subset of the SFT dataset.The reward assigns 1 for a verified answer, 0.0 for a failed verification, and −1 for an invalid answer format.Source: §4 · pp. 7–8
  • DPO achieves 47.2 on average, including 46.1 on college_math and 40.0 on high_school_league.The paper describes DPO as aligning models directly with preference data without explicit reward modeling.Source: §4 · pp. 7–8
  • PPO also averages 47.2, with 69.9 on math-hard and 44.2 on high_school_league, but scores 26.7 on AIME24.PPO adjusts policies within constrained updates while balancing exploration and exploitation.Source: §4 · pp. 7–8
  • REINFORCE++ scores 30.0 on AIME24 but averages 45.9, reflecting stronger task-specific performance than overall consistency.The method is described as reducing variance and improving convergence speed relative to classic REINFORCE.Source: §4 · pp. 7–8
  • 47.2 average performance is achieved by both DPO and PPO, making them the strongest of the three evaluated RL algorithms.REINFORCE++ reaches an average score of 45.9.Source: §4 · p. 8, Table 3 · p. 8

5 Experiments

The experiments evaluate Long-COT scaling and training choices on reasoning, general benchmarks, and out-of-distribution tasks. RedStar combines Long-COT data and reinforcement learning to improve reasoning while retaining broad capabilities.

5.1 Experiment Settings

  • The 32B experiment uses 4k math-derived and 16k code-derived Long-COT samples, with DPO selected over PPO because PPO required more resources despite comparable results.Context parallelism supports training at 16K context length.Source: §5.1 · p. 8

5.2 Main Results

  • 58.3 average score is achieved by RedStar-DPO, the strongest reported reasoning result, while RedStar models outperform Qwen models across most tasks.RedStar-DPO combines reinforcement learning with Long-COT tuning.Source: §5.2 · p. 9
  • 55.7 average score for RedStar-code-math exceeds RedStar-code’s 43.6, indicating cross-domain benefits from combining math and code Long-COT datasets.RedStar-code also surpasses Qwen2.5-32B-Instruct’s 37.9 average score.Source: §5.2 · p. 9
  • 83.5 on Math-hard and 53.3 on Aime24 show RedStar-DPO’s strongest task-specific results on difficult multi-step reasoning tasks.The results support specialized reinforcement learning and task-specific fine-tuning for particular reasoning domains.Source: §5.2 · p. 9
  • RedStar achieves comparable performance to closed-source and open-source models on Chinese graduate mathematics OOD tests despite training on English datasets.The evaluation uses human-verified results on an OOD Chinese mathematics dataset.Source: §5.2 · p. 9, Table 6 · p. 10

5.3 Impact on General Foundational Capability

  • 68.8 overall average score places RedStar above Qwen2.5-32B-Instruct’s 68.1 and Qwen-QwQ-32B’s 62.8 across general benchmarks.RedStar leads in reasoning and factuality, while remaining competitive in STEM, long-text generation, and SedarEval; CommonSense remains weaker than Qwen2.5-32B-Instruct.Source: §5.3 · pp. 10–11, Table 5 · p. 10

6 One More Step: Applying Slow-Think to MLLM

The multimodal study tests whether Long-COT improves geometric reasoning in vision-language models. RedStar-Geo achieves strong performance with a small GeoQA-derived instruction set and generalizes to OOD geometric benchmarks.

Evaluation Setup

  • The study filters non-reflective responses and excessively long-waiting samples before evaluating GeoQA, MathVista-GEO, and Geometry3K.This procedure is intended to improve response quality and model efficiency.Source: §6 · p. 11

Multimodal Results

  • A small GeoQA-derived Long-COT instruction set enables RedStar-Geo-8B to surpass GPT-4o/V and comparable models trained on the same dataset size and base model.The study uses 3,499 GeoQA training samples and selects Intern2VL-8B as the base model.Source: §6 · p. 11
  • Less than 1/50th of Geo170K’s instruction-set size yields similar performance through multimodal Long-COT instruction tuning.The comparison includes models trained with substantially larger datasets and Math-LLaVA’s multimodal Short-CoT data.Source: §6 · pp. 11–12
  • Multimodal Long-COT supports finer-grained reflection on visual elements and superior generalization on MathVista-GEO and Geometry3K OOD datasets.These benchmarks include geometric proofs and richly illustrated geometry problems.Source: §6 · pp. 11–12

7 Related Work

Related work frames slow-thinking systems as deliberate reasoning models built through CoT, reinforcement learning, search, distillation, and iterative improvement. RedStar instead emphasizes Long-COT data and model scaling, including multimodal reasoning.

Slow-Thinking Reasoning Systems

  • Slow-thinking systems target deliberate System-2-style reasoning rather than direct next-token responses, with applications spanning code, translation, medicine, and multimodal understanding.Prior methods include CoT prompting, reinforcement learning, iterative improvement, and search-based reasoning.Source: §7 · p. 12

Positioning

  • Earlier approaches emphasize tree search, data synthesis, and distillation, whereas RedStar studies model scaling, data scaling, and reinforcement learning for Long-COT tuning.The paper also extends the approach to multimodal geometric reasoning.Source: §7 · p. 12

Long Context Synthesis

  • Long-context synthesis research often concatenates or retrieves documents, but complex long reasoning paths remain less studied.RedStar focuses on data-source selection, RL sample construction, and Long-COT scaling for slow-thinking systems.Source: §7 · pp. 12–13

8 Conclusion and Future Work

The paper concludes that Long-COT can improve reasoning with limited data, preserve general capabilities, scale through reinforcement learning, and benefit vision-language models. Future work targets more accessible data synthesis and broader evaluation.

Conclusion

  • Long-COT produces significant, scale-dependent reasoning gains with minimal samples and does not hinder general task performance.Reinforcement learning further enhances scalability, while VLM applications improve multimodal performance.Source: §8 · p. 13

Future Work

  • Future work will synthesize high-quality Long-COT datasets using only instruction-tuned models and test generalization on additional complex benchmarks.The stated aim is more accessible, scalable, and automated data generation.Source: §8 · p. 13

A Bitter Failure: Step-by-Step Verification

Step-by-step verification did not produce substantial gains over a no-verifier baseline, especially on complex Long-CoT problems. The results indicate that current critics and verification rules remain insufficient for scalable oversight.

  • The method combines compact-solution extraction, rejection sampling, first-error-step detection, and five sampling iterations to produce strict and loose datasets.The strict version excludes detected errors, whereas the loose version permits at most two detected errors.Source: A Bitter Failure: Step-by-Step… · p. 18
  • Verification with strict and loose rules did not significantly outperform the no-verifier baseline across most tasks, particularly on complex problems.Even o1-mini, described as the strongest critic tested, struggled to identify errors in challenging Long-CoT solutions.Source: A Bitter Failure: Step-by-Step… · p. 18
  • Generalizing multi-step error verification across diverse tasks remains unresolved, while critic sensitivity to intricate reasoning patterns requires further refinement.The authors attribute limited gains partly to the difficulty of detecting errors in complex reasoning and to Long-CoT’s own reduced susceptibility to some errors.Source: A Bitter Failure: Step-by-Step… · p. 18
  • The current verification implementation did not yield the expected reasoning improvements, motivating more robust oversight methods and higher-quality datasets.Source: A Bitter Failure: Step-by-Step… · p. 18

B RL Experimental Setups

The evaluation reports RedStar’s performance across STEM, reasoning, common-sense, long-generation, SedarEval, and instruction-following benchmarks. Results show strong reasoning and broad task performance, alongside weaknesses in very long generation and instruction following.

B RL Experimental Setups

C.1 STEM

  • RedStar achieves a 79.2 STEM average, below Qwen2.5-32B-Instruct (80.1) but above Qwen-QwQ-32B (71.7).It is competitive on MMLU and CMMU, while MMLU Pro Engineering remains weaker at 48.9 versus 57.4 for Qwen2.5-32B-Instruct.Source: §C.1 · p. 19, Table 10 · p. 19

C.2 Reasoning

  • RedStar scores 73.1 on reasoning benchmarks, slightly above Qwen-QwQ-32B (73.0) and above Qwen2.5-32B-Instruct (65.5).Source: §C.2 · p. 19

C.3 Common Sense

C.4 Long Generation

  • RedStar remains competitive on long-generation tasks, especially heuristic generation, summarization, and open-ended QA, but quality degrades beyond 4000 words.The benchmark covers open-ended QA, summarization, chat, text completion, and heuristic generation.Source: §C.4 · pp. 20–21, Table 14 · p. 21

C.5 SedarEval Evaluation

  • RedStar performs exceptionally across SedarEval tasks, particularly mathematics and coding, outperforming Qwen-QwQ-32B and Qwen2.5-32B-Instruct on difficult questions.SedarEval uses self-adaptive rubrics across long-tail knowledge, mathematics, coding, and logical reasoning.Source: §C.5 · p. 21, Table 15 · p. 21

C.6 Instruct Following

  • RedStar scores 55.5 on IFEval, above Qwen-QwQ-32B (41.4) but below Qwen2.5-32B-Instruct.The comparison indicates that slow-thinking models still lag significantly behind Qwen2.5-32B-Instruct on instruction following.Source: §C.6 · p. 21, Table 16 · p. 22

C.7 Scaling Results on General Tasks

Scaling Long-CoT data has opposite effects on reasoning and general alignment for smaller models. As data increases, the 7B model loses substantially on Hellobench and Sedareval, while the 14B model declines more steadily.

  • As Long-CoT data scales from 0 to 1000k, the 7B model drops from 83.23 to 37.60 on Hellobench and from 57.55 to 26.04 on Sedareval.The passage contrasts these declines with positive improvements in reasoning performance.Source: §C.7 · p. 22
  • The 14B model shows a steadier performance decrease on general alignment tasks as Long-CoT data scales.The authors attribute the difference to larger models’ improved stability and generalization, while hypothesizing specialization and catastrophic forgetting in smaller models.Source: §C.7 · p. 22

Related papers

Loading 2501.11284v1…