Source-linked AI summary
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, Chong Luo
TL;DR
The paper asks whether reproducible rule-based RL can develop reasoning in smaller models and generalize beyond synthetic logic training. It trains a 7B model with Logic-RL on controllable Knights and Knaves puzzles using shortcut-resistant rewards and evaluates broader reasoning and mathematical transfer. After 5,000 puzzles, the model improves on AIME and AMC and exhibits reasoning behaviors including reflection, verification, and summarization.
Problem
DeepSeek-R1 demonstrates reasoning abilities but does not release its training code or dataset, leaving reproducibility and smaller-model questions open.
Method
Logic-RL trains a 7B model with rule-based RL on procedurally generated logic puzzles using REINFORCE++, a system prompt, stringent format rewards, and reward modifications.
Results
125% AIME and 38% AMC improvement against the base model followed training on 5,000 procedurally generated logic puzzles.
Takeaways & Limitations
The model develops reflection, verification, summarization, and other reasoning patterns, while reported findings favor RL generalization over SFT memorization.
Takeaways & Limitations
The findings are based on a small-scale logic dataset, and generalization to large-scale real-world mathematical or coding scenarios remains unexplored.
Abstract
from arXiv · showhide
Inspired by the success of DeepSeek-R1, we explore the potential of rule-based reinforcement learning (RL) in large reasoning models. To analyze reasoning dynamics, we use synthetic logic puzzles as training data due to their controllable complexity and straightforward answer verification. We make some key technical contributions that lead to effective and stable RL training: a system prompt that emphasizes the thinking and answering process, a stringent format reward function that penalizes outputs for taking shortcuts, and a straightforward training recipe that achieves stable convergence. Our 7B model develops advanced reasoning skills-such as reflection, verification, and summarization-that are absent from the logic corpus. Remarkably, after training on just 5K logic problems, it demonstrates generalization abilities to the challenging math benchmarks AIME and AMC.
1 Introduction
The paper addresses reproducibility gaps in rule-based RL reasoning by using controllable logic puzzles and a framework designed to prevent shortcut solutions. With 5,000 puzzles, the 7B model improves on AIME and AMC while exhibiting broader reasoning behaviors.
- DeepSeek-R1's missing training code and dataset leave open whether smaller models can acquire similar reasoning abilities and how to reproduce them.
- Procedurally generated Knights and Knaves puzzles provide controllable difficulty and straightforward rule-based reward verification for studying reasoning.
- Logic-RL combines REINFORCE++, DeepSeek-R1-inspired rewards, a practical system prompt, and stringent format rewards to prevent shortcut solutions.
- 5,000 logic puzzles yielded 125% AIME and 38% AMC improvement for the 7B model against the base model.
- The study reports that longer responses do not guarantee better reasoning, language mixing hinders reasoning, and increasing thinking tokens helps.
- The study reports that SFT memorizes whereas RL generalizes, cold start is not necessary, and curriculum learning outperforms random shuffling under fixed curation ratios.
2 Method
Logic-RL trains reasoning on procedurally generated Knights and Knaves puzzles using rule-based rewards, structured prompting, and a modified REINFORCE++ recipe. The method controls puzzle difficulty, verifies answers deterministically, and targets shortcut-resistant reasoning behavior.
- Puzzle Dataset: K&K puzzles provide procedurally generated, controllable training data with unique ground-truth answers for rule-based verification.Difficulty varies by character count and Boolean-operator complexity, enabling curriculum and out-of-distribution evaluation.
- Rule-Based Rewards: The reward combines a format score and an answer score to penalize shortcuts and reward correct, extractable solutions.Format correctness receives 1 or -1, while a fully matching answer receives 2; iterative rule refinement addressed common hacking behaviors.
- Structured Prompting: The system prompt requires reasoning inside <think> tags and a readable final conclusion inside <answer> tags.Including a <think> tag at the prompt end reduces the base model’s difficulty in following the required structure.
- Rule-Based Rewards: Early reward designs were hacked through skipped or misplaced reasoning, repeated guessing, irrelevant text, and other format violations.The authors refined rules to require each tag exactly once, in sequence, with genuine reasoning and an extractable conclusion.
- RL Algorithm and Schedule: Training uses a modified REINFORCE++ baseline with KL regularization, discounted returns, and fixed hyperparameters over mixed-complexity puzzles.The model is trained for 3600 steps at learning rate 4 × 10^-7 and temperature 0.7, with γ set to 1 in the experiments.
- RL Algorithm and Schedule: The modified KL estimation keeps the KL estimate non-negative, while the training process develops exploration, verification, and summarization before final answers.The per-token KL divergence is used as a reward penalty, and the reported training regimen produces stable reasoning patterns.
3 Experiment
The experiments select Qwen2.5-7B-Instruct-1M as the baseline after comparing Qwen2.5 variants and evaluate Logic-RL on K&K puzzles across difficulty levels. The trained model generalizes beyond its 3–7-person training distribution to 8-person puzzles while developing longer, more reflective responses.
- Baseline selection: Qwen2.5-Math-7B frequently generated Python code blocks that conflicted with the experiment’s strict formatting requirements.Removing system prompts and penalizing specific markdown styles did not fully resolve this behavior.
- Baseline selection: Qwen2.5-7B-Instruct-1M was selected as the baseline because it achieved slightly higher test accuracy than the base model.Qwen2.5-7B-Base and Qwen2.5-7B-Instruct showed nearly identical RL training metrics, including validation accuracy, response-length growth, and reward curves.
- Generalization: The model generalized to 8-person K&K puzzles despite training on fewer than 5,000 synthetic puzzles containing only 3–7 people.This evaluates performance on an out-of-distribution scenario beyond the training distribution.
- Evaluation: Table 2 compares reasoning and general models on K&K logic puzzles across varying difficulty levels.The supplied caption identifies the models, task, and difficulty comparison but does not report specific table outcomes.
- Reasoning dynamics: After 1k RL steps, average output length increased from 500 tokens to 2000 tokens, alongside reflection and exploration of alternative solutions.The passage describes this increase as nearly linear and associates the emerging behaviors with handling more complex tasks.
4 Research Questions
The experiments compare reinforcement-learning algorithms, reasoning-related tokens, the emergence of reasoning behaviors, out-of-distribution transfer, memorization, curriculum learning, and response length. Results favor REINFORCE++, show mixed effects for thinking tokens and language mixing, strong transfer to AIME and AMC, better RL generalization than RFT, limited curriculum impact, and no reliable benefit from longer outputs.
- RQ 1: RL algorithms: REINFORCE++ outperformed GRPO across nearly all metrics, while PPO achieved higher accuracy and reward but was 138% slower than REINFORCE++.GRPO exhibited the weakest performance among the three evaluated algorithms.
- RQ 2: Thinking tokens and language mixing: Responses containing "verify" and "re-evaluate" scored significantly higher, whereas certain tokens from other languages generally received lower rewards.The experiments analyzed answer rewards for responses containing the displayed tokens.
- RQ 2: Thinking tokens and language mixing: "Recheck" diminished reasoning ability, and "re-evaluate" produced much higher answer scores than "reevaluate".The paper links the difference partly to the models’ differing exposure to these forms in pretraining data.
- RQ 3: Aha moment: Reflective and exploratory behaviors emerged organically during RL, but their word frequencies increased steadily without a distinct sudden "aha moment."The tracked behaviors included reflection, verification, summarization, and exploration.
- RQ 5: SFT versus RL: RL achieved higher test accuracy with minimal or negative memorization-score increases, whereas RFT’s accuracy gains accompanied rapidly increasing memorization.The comparison uses the Test accuracy–LiMem(f; Tr) relationship to assess generalization.
- RQ 6: Curriculum learning: Curriculum learning yielded slightly higher intermediate test scores, but early differences were statistically negligible and its practical necessity was not conclusively supported.The analysis used rolling averages with a window size of 5.
- RQ 7: Response length: Increasing response length alone did not improve validation accuracy or reward, while the positive model improved both as its response length decreased.The findings suggest length changes are training-dynamics correlates rather than reliable causal drivers of reasoning improvement.
5 Discussion and Future Work
The discussion identifies limitations in the study’s small-scale logic-data setting and proposes future work on efficiency, training stability, mixed-language reasoning, and less constrained reasoning formats. These directions aim to test broader robustness and improve the scalability of RL-based reasoning training.
- Limitations: The findings are based on a small-scale logic dataset, so generalizability to large-scale real-world mathematical or coding scenarios remains unexplored.The paper calls for evaluation on more diverse and complex datasets.
- Future work: Chain-of-thought responses can expand by up to four times after RL, motivating methods that transform long responses into shorter, more efficient formats.The proposed direction targets token efficiency and long-context-friendly training.
- Future work: Eliminating KL constraints may benefit training from strong foundation models, while higher initial temperature may provide a more diverse starting point.The effects of the supervised fine-tuning stage remain an open investigation.
- Future work: Chinese tokens appeared frequently in English thinking sections despite English-only training data, motivating investigation of whether token switching can aid internal reasoning.The paper presents favorable hidden states under the RL scheme as one hypothesis.
- Future work: It remains open whether unconstrained or latent reasoning representations could outperform the explicit <think>...</think> format.The paper suggests suitable incentives might lead models to develop their own internal reasoning representation.
6 Related Work
Related work frames LLM reasoning around chain-of-thought decomposition and search-based planning, then situates Logic-RL among post-training methods that use supervised fine-tuning, preference optimization, or reinforcement learning on reasoning data.
- Large Language Model Reasoning: Chain-of-Thought reasoning breaks complex problems into manageable steps to enhance logical reasoning.The related-work discussion also describes MCTS as a tree-based exploration–exploitation method for model-based planning.
- LLM post-training for reasoning: Recent LLM post-training methods improve reasoning through fine-tuning or reinforcement learning on specialized datasets containing reasoning examples and chain-of-thought explanations.The discussion places DPO, PPO, GRPO, and REINFORCE++ within this broader post-training landscape.
7 Comparion Between Base and Instruct Model
Base and instruct models follow nearly identical RL training trajectories, while differing in how quickly response length grows and in validation accuracy. The base model increases thinking compute faster, whereas the instruct model maintains slightly higher validation accuracy.
- RL training produces nearly identical accuracy, response-length, and reward trajectories for base and instruct models.
- The base model’s response length grows faster than the instruct model’s during RL training.
- The instruct model consistently maintains slightly higher validation accuracy while the base model rapidly increases thinking compute.
8 Qualitative Analysis of Emergent Reasoning
RL training produces emergent reasoning behaviors beyond the supplied logic-puzzle patterns, including verification, backtracking, formal implication use, and occasional language switching. These behaviors are illustrated through qualitative examples from the trained model.
- Hesitation and Self-Verification: The model develops hesitation and self-verification before producing final answers.It re-checks prior steps after expressing uncertainty, a behavior absent in pre-training.
- Multi-Path Exploration & Backtracking: The model explores multiple solution paths and backtracks to test consistency, resembling human problem-solving.
- Formula Application: The model applies the formal “If P, then Q” implication rule when solving Knights and Knaves puzzles.The cited example states that the implication is false only when P is true and Q is false, despite no such training data.
- Occasional Language Switching: The model occasionally switches languages during reasoning while keeping the final answer in English.The passage describes Chinese tokens appearing in some thinking segments before the solution returns to English.