Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning

Mathematical Reasoning

• 86.7 on AIME 2024 matches o3-mini-high and significantly exceeds o1 and DeepSeek R1; on BeyondAIME, Seed1.5-Thinking surpasses o1 and R1 but trails o3 and Gemini pro 2.5.BeyondAIME uses newly curated human-expert problems designed to reduce solving through memorization or guessing.Source: §1 · p. 2

Competitive Programming

• On the most recent 12 Codeforces contests, Seed1.5-Thinking outperforms DeepSeek R1 on pass@1 and pass@8, though it trails o3.pass@k indicates whether a model solves a problem within k attempts; pass@8 is reported for stability and closer alignment with user submissions.Source: §1 · p. 2

Science

• 77.3 on GPQA is close to o3-level performance, attributed largely to generalization from mathematical training rather than increased domain-specific science data.Source: §1 · p. 2

Non-reasoning Tasks

• Seed1.5-Thinking achieves an 8.0% overall rise in positive user feedback against DeepSeek R1 across diverse real-world scenarios.The evaluation used human comparisons on a test set designed to replicate real-world user needs.Source: §1 · p. 2

Data

• RL data spans STEM, code, logic, and non-reasoning tasks; logic data improves ARC-AGI, while math data generalizes broadly across tasks.For SFT, reasoning models use chain-of-thought data; preliminary experiments found that too much non-CoT SFT data can reduce exploration.Source: §1 · p. 2

RL Algorithm

• RL runs can differ by up to 10 points; VAPO [5] and DAPO [6] target instability in actor-critic and policy-gradient training, enabling robust, consistent optimization.VAPO is described as state of the art for actor-critic methods, while DAPO is state of the art for policy-gradient approaches without critic models.Source: §1 · pp. 2–3

RL Infrastructure

• An asynchronous rollout system with prioritized sample pools achieves 3× faster iteration cycles than synchronous frameworks.It also supports mixed-precision training and automatic fault recovery for large-scale RL runs.Source: §1 · p. 3

2.1 RL Training Data

• Reasoning ability primarily comes from verifiable problems with definitive answers and generalizes to non-verifiable problems without definitive answers.Source: §2.1 · p. 3

2.1.1 Verifiable Problems

• Verifiable problems include STEM questions paired with answers, coding problems with unit tests, and logic reasoning amenable to automated verification.Source: §2.1.1 · p. 3

STEM Data

• Several hundred thousand competition-grade problems, over 80% mathematical, are drawn from open, public, and proprietary sources; cleaning and augmentation yield 100k STEM training problems.Seed-Verifier evaluates response correctness during training.Source: STEM Data · p. 3
• Cleaning removes incomplete, unclear, or overly simple problems, checks suspect reference answers with reasoning models and human experts, and converts multiple-choice or math items to reduce guessing.The team also modifies some math problems so their answers are integers whenever possible.Source: STEM Data · p. 3
• Coding tasks prioritize competitive programming and require clear specifications, unit tests, and checker scripts; offline sandbox evaluation correlates strongly with official verdicts.Difficulty filtering selects appropriately challenging problems, and an in-house sandbox executes model-generated code for consistent, high-throughput feedback during RL.Source: STEM Data · pp. 3–4
• Logic data covers 22 tasks with configurable generators and verifiers; training difficulty adjusts to model performance, and about 10k puzzle problems are generated for RL.Verifiers assess correctness and can serve as reward functions in RL pipelines.Source: STEM Data · p. 4

2.1.2 Non-verifiable Problems

• Non-verifiable prompts cover creative writing, translation, knowledge QA, and role-playing across diverse domains, drawing on Doubao-1.5 Pro RL training data.Source: §2.1.2 · p. 4
• Prompts with low reward-score variance, low difficulty, or gains already surpassed during Doubao 1.5 Pro RL training are removed; pairwise rewards compare sample quality.The paper reports that overoptimizing the excluded samples can prematurely collapse exploration and reduce performance.Source: §2.1.2 · p. 4

2.2 Advanced Math Benchmark

• To address AIME’s 30-problem annual release and high-variance evaluations, specialists curate 100 original or adapted, nonduplicate problems at least as difficult as AIME’s hardest, with integer answers.Problem adaptations use structural modifications and scenario reconfigurations, and answers are designed not to be trivial values stated in the prompt.Source: §2.2 · p. 4

3.1 Reward Modeling for Verifiable Problems

• For verifiable questions, LLM-based Seed-Verifier and Seed-Thinking-Verifier provide a generalized alternative to rule-based rewards, using human-crafted principles or trained reasoning paths to judge answer equivalence.Seed-Verifier evaluates the question, reference answer, and generated answer; Seed-Thinking-Verifier is optimized as a verifiable task alongside mathematical reasoning tasks.Source: §3.1 · p. 5
• Its detailed reasoning helps address reward hacking, inconsistent equivalent-answer judgments, and difficult corner cases.The Seed-Verifier may return inconsistent judgments when equivalent answers differ in format, while detailed analysis supports more consistent decisions.Source: §3.1 · p. 5
• Seed-Thinking-Verifier makes more accurate judgments than Seed-Verifier on difficult cases, though its reasoning consumes significant GPU resources.Table 1 reports accuracy for the two verifier types; the supplied passages do not provide its numerical results.Source: §3.1 · p. 5, Table 1 · p. 5

3.2 Reward Modeling for Non-verifiable Problems

• For non-verifiable problems, the reward model uses human-preference data from Doubao 1.5 Pro and pairwise generative scoring from to improve mixed-training stability.It compares two responses and uses the probability of “YES” or “NO” as the reward, helping limit outlier scores and conflicts with verifier rewards.Source: §3.2 · p. 6

4.1 Supervised Fine-Tuning

• SFT uses 400k instances—300k verifiable and 100k non-verifiable—and produces more readable, less hallucinatory, less harmful outputs than starting RL from a base model.The non-verifiable data comes from Doubao-Pro 1.5 SFT data and covers creative writing, knowledge-based QA, safety, and function calling.Source: §4.1 · p. 6
• An iterative workflow combines expert-created reasoning responses, model training, and Seed-Verifier rejection sampling to produce detailed long-CoT trajectories across mathematics and other domains.The workflow also applies to coding, logic puzzles, and creative writing.Source: §4.1 · p. 6

4.2 Reinforcement Learning

• The unified RL framework combines verifier-scored data, reward-model-scored general data, and hybrid data scored by both mechanisms.Verifiable data supports direct validation against known criteria, while reward-model scores reflect alignment with human preferences.Source: §4.2 · p. 6
• To address value-model bias and sparse rewards, value pretraining aligns the value model with the fixed policy, while decoupled GAE separates value and policy estimation parameters.The cited prior-work techniques are [10]; decoupled GAE uses λvalue = 1.0 and λpolicy = 0.95.Source: §4.2 · pp. 6–7
• Length-adaptive GAE varies λpolicy with response length; Clip-Higher encourages low-probability-token exploration, token-level loss weights tokens, and positive-example LM loss improves learning from positive samples.Length-adaptive GAE aims to distribute TD errors more uniformly across short and long sequences.Source: §4.2 · p. 7
• Online Data Distribution Adaptation changes the stationary RL prompt distribution into an adaptive one to reduce interference across domains and support more balanced improvement.The cited interference sources include difficulty disparities and reward-hacking risk.Source: §4.2 · p. 7

5.1 Framework

• HybridFlow [11] runs on a Ray [12] cluster, where a single controller invokes Worker Group APIs for SPMD training and generation workloads.The dataloader and RL algorithm run in a single-process Ray Actor.Source: §5.1 · pp. 7–8
• SRS addresses long-CoT generation stragglers by deploying standalone streaming-compute units to shift system constraints from memory-bound to compute-bound.The hybrid engine co-locates models to avoid GPU idle time when switching between training and generation.Source: §5.1 · p. 8

5.2 Streaming Rollout System

• Streaming rollout decouples model evolution from runtime execution, using α to control on-policy samples and asynchronous continuations of off-policy generations from versioned snapshots.α is the proportion of samples generated on-policy using the latest model version; the remaining 1-α uses off-policy rollouts.Source: §5.2 · p. 8
• For environment interaction, the system combines FP8 policy networks with error-compensated range scaling, TP/EP/SP parallelism, and load-aware CUDA kernel auto-tuning.Dynamic expert assignment and sequence parallelism for context chunking address token imbalance in MoE systems.Source: §5.2 · p. 8

5.3 Training System

• Training combines TP/EP/CP with FSDP, applying TP/CP to attention layers and EP to MoE layers.KARP rearranges sequences within a mini-batch to balance effective sequence lengths across micro-batches.Source: §5.3 · p. 8
• Layer-wise recomputation [15], activation and optimizer offload, and AutoTuner’s profile-based estimates of memory and performance support efficient training configurations.These memory optimizations support larger micro-batches that overlap FSDP communication overhead.Source: §5.3 · p. 8
• ByteCheckpoint supports resuming from different distributed configurations with minimal overhead, enabling elastic task training to improve cluster efficiency.Source: §5.3 · p. 8

6.1 Auto Evaluation Results

• 86.7 on AIME 2024 matches OpenAI’s o3-mini-high, while 77.3% accuracy on GPQA is close to o3-mini-high.Seed1.5-Thinking nearly matches Gemini 2.5 Pro but trails o3-mini-high on Codeforces and lags o3-level performance on AIME 2025 and BeyondAIME. Its weaker SimpleQA results concern a memory-oriented benchmark more associated with pretrained model scale than reasoning.Source: §6.1 · p. 9

6.2 Human Evaluation Results

• 8.0% overall win ratio against Deepseek R1 across evaluated sessions indicates stronger alignment with human-centric preferences across creative writing and humanities knowledge elaboration.Expert evaluators rated multi-round sessions against predefined rubrics on a 0–4 scale, assigning per-round scores and session-level win/loss outcomes.Source: §6.2 · pp. 9–10

6.3 Effects of pre-train models

• RL algorithms retain consistent rankings across model sizes and architectures, supporting Qwen-32B as a proxy for investigating RL algorithms.Seed-150B-MoE exhibits the same ranking despite differing from Qwen-32B in both MoE-versus-dense architecture and size.Source: §6.3 · p. 10
• RFT initialization saturates more quickly during training but ultimately achieves lower performance than training without RFT.The ablation examines whether initializing RL with a rejection fine-tuning model affects outcomes.Source: Table 3 · p. 10, §6.3 · p. 10

7 Related Work

• Test-time scaling and extended CoT reasoning have catalyzed a paradigm shift, helping LLMs excel at complex math and coding tasks, while scalable RL methodologies remain largely obscure [23] [22].The cited examples include OpenAI’s o1 and DeepSeek’s R1; large-scale reinforcement learning can elicit behaviors such as self-verification and iterative refinement.Source: §7 · p. 10
• Seed1.5-Thinking presents its performance approach through data, RL algorithms, and RL infrastructure.Source: §7 · p. 11

8 Conclusion

• The model performs well across both reasoning tasks and non-reasoning tasks.The conclusion describes Seed1.5-Thinking as a superb reasoning model.Source: §8 · p. 11
• Seed1.5-Thinking achieves 86.7% on AIME24, 74.0% on AIME25 and 55.0% on Codeforces.These results are attributed to advanced RL techniques for improving thinking ability stably and reliably.Source: §8 · p. 11
• The authors plan to investigate more efficient RL recipes.Source: §8 · p. 11
• The authors plan to explore more challenging tasks with thinking mode to push the boundary of model’s intelligence.Source: §8 · p. 11
• General reward modeling with accuracy comparable to a verifier is identified as a compelling research direction.Source: §8 · p. 11

A Case Study on Verifier

• Seed-Verifier struggles with samples that have complex answers, while Seed-Thinking-Verifier provides accurate judgments through step-by-step analysis.Source: A Case Study on Verifier · p. 15
• The paper attributes Seed-Thinking-Verifier’s flexibility and broad domain generalization to its detailed thinking process.Source: A Case Study on Verifier · p. 15

• The creative-writing examples span Chinese and English and are presented to demonstrate the model’s proficiency.Source: B Case Study on Creative Writing · p. 15
• Each example contains the original user prompt, the model’s chain of thought, and its final response.Source: B Case Study on Creative Writing · p. 15
• Tables 6, 7, and 8 present three cases on creative writing.Source: Table 6 · p. 17, Table 7 · p. 18, Table 8 · p. 19