LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning

• Unlike prior work centered on propositional logic or formal languages, LogicPro draws on algorithm problems with explicit input-output relations and recursive or iterative structures.Source: §1 · p. 2
• The pipeline combines algorithm problems with test cases, executes Python solutions for answers and intermediate values, then uses those values to guide reasoning synthesis.Source: §1 · p. 2
• The dataset is described as difficult and scalable: models perform worse on it than on other baseline data, and adding problems and test cases can expand it.Source: §1 · p. 2
• LogicPro produces 540K examples from 2,360 algorithm problems, with standard answers and high-quality reasoning processes.Source: §1 · p. 2
• LogicPro improves multiple models on BBH27, LogicBench, DROP, AR-LSAT, and GSM8K, outperforming a wide range of existing datasets.Source: §1 · p. 2

2.1 Data Collection

• The collection uses 2,360 official LeetCode problems and GPT-4-generated test cases, with up to 300 cases compiled per problem.Source: §2.1 · p. 2

2.2 Step 1: Construct Complex Reasoning Problem

• The first stage generated 699K text problems and retained 595K after consistency and solvability checks, including filtering 5K+ inconsistent and 98K unsolvable items.Source: §2.2 · pp. 3–4

2.3 Step2: Obtain Intermediate Variable from Program

• The code stage generates test-specific Python solutions that print final results and key intermediate variables; execution checks yield 544K data points with answers and intermediate outputs.Source: §2.3 · p. 4

2.4 Step3: Program-Guided Reasoning

• The final stage uses text reasoning problems and code intermediate-variable outputs to guide reasoning generation, yielding 540K question-answer pairs.Source: §2.4 · pp. 4–5

3.1.1 Training

• Training mixes synthetic and baseline data with either 100K OpenHermes general-domain data or LogiCoT logical-reasoning data, across small- and large-scale models.The main experiments use Qwen-7B, Llama3-8B, Qwen2-72B, and Llama3-70B.Source: §3.1.1 · p. 5

3.1.2 Evaluation

• Ten reasoning benchmarks are selected; all are out-of-domain for LogicPro, although LogicBench’s training and test sets are in-domain.Seven benchmarks are reported in the main text, with complete results in Appendix A.2 Table 6.Source: §3.1.2 · p. 5
• Evaluation uses greedy decoding with a 4096-token generation limit and Qwen-2.5-72B to score model outputs.Inference is deployed with vLLM.Source: §3.1.2 · p. 5

3.2 Main Results

• 2.3%-4.7% average-performance gains over the previous best baseline on BBH, and at least 10% over general baseline data, span model types and scales.The weighted average also improves versus baselines by 3.9%-8% for Qwen2 7B, 3.1%-9.1% for Qwen2 72B, 2.5%-8.8% for Llama3 8B, and 1.6%-9.7% for Llama3 70B.Source: §3.2 · p. 6, Table 1 · p. 6
• LogicPro leads on LogicBench except when LogicBench itself is the training set; three models trail other baselines on FOLIO, while DROP and AR-LSAT show multi-foundation advantages.The authors suggest first-order-logic baselines may be closer to FOLIO’s data distribution; synthetic-data choice has relatively minor impact on GSM8K.Source: §3.2 · p. 6

4.1 Ablation Study

• Adding intermediate-variable guidance improves BBH by 2.68% and 1.04%, and Average by 2.2% and 1.1%, over direct distillation; question synthesis alone also raises BBH and Average.The question-synthesis comparison reports BBH gains of 7.21% and 7.34% and Average gains of 6.8% and 6.6% for Qwen2-7B and Llama3-8B, respectively.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 Performance Gain on All Baseline Data

• Adding LogicPro to all baseline data improves BBH by 5.6% and 4.7%, and Average by 5.2% and 3.6%; the authors frame it as complementary, not a replacement.Table 3 compares the mixed baseline data with and without LogicPro.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Data Scaling Analysis

• Data scaling shows a steady upward trend, whereas increasing algorithm-problem counts produces BBH and Average fluctuations attributed possibly to distribution differences among problems.The figure compares scaling total data, the number of algorithm problems, and the number of test cases.Source: §4.3 · p. 7, Figure 6 · p. 8
• Test-case scaling generally trends upward, but performance declines beyond 80%; with fixed problems, diminishing returns and limited GPT-4-generated test-case quality constrain gains.The authors recommend improving test-case quality while increasing the number of algorithmic problems to address the expansion bottleneck.Source: §4.3 · pp. 7–8

4.4 Difficulty Comparison with Baseline Data

• Four open-source models average 45.6% accuracy on LogicPro, versus over 50% on existing baselines and over 70% on some datasets.The comparison samples 5K examples per dataset and uniformly samples LogicPro problems.Source: §4.4 · p. 8, Table 4 · p. 8

5.1 Synthetic Reasoning Data

• Synthetic data for mathematical reasoning uses problem-driven techniques or knowledge bases and concept graphs, while code-data work includes self-instruction, code evolution, and open-source-code extraction.Examples include evol-instruct, problem restatement, back-translation, and few-shot generation for mathematics, and Code Alpaca, WizardCoder, and Magicoder for code.Source: §5.1 · pp. 8–9
• Complex logical-reasoning data synthesis is described as less researched, with prior approaches drawing on formal logic, propositional logic, and symbolic methods.Examples include synthetic data based on formal logic and work covering propositional-logic benchmarks or logical corpora.Source: §5.1 · p. 9, §5.2 · p. 9

5.2 Symbolic Reason Enhances LLM Reason

• LogicPro uses LeetCode-style algorithm problems as a distinct source of reasoning patterns, rather than relying on propositional logic or formal languages.The paper identifies explicit input-output relationships, recursive and iterative structures, and data-structure operations as characteristic patterns.Source: §5.1 · p. 9, §5.2 · p. 9

6 Conclusion and Future Work

• LogicPro constructs 540K examples from 2,360 algorithm problems and improves performance across BBH27, LogicBench, DROP, AR-LSAT, and GSM8K against a wide range of existing datasets.The method uses LeetCode-style algorithm problems and solutions to synthesize complex logical-reasoning data.Source: §6 · p. 9
• Future work may expand the dataset with problems from Luogu, ACM competitions, and online judges; execution results and intermediate variables may also offer signals for reinforcement learning.Source: §6 · p. 9

Limitations

• The method explores synthesizing complex logical reasoning data from a LeetCode-style algorithmic approach.Source: Limitations · pp. 9–10
• More comprehensive and diverse test cases are a future direction, with advanced test-generation techniques expected to improve synthetic-data quality and generalization.Source: Limitations · p. 10
• The current study uses 2,360 LeetCode problems, while problems from Luogu, ACM competitions, and other Online Judge platforms remain potential sources for broader coverage.Source: Limitations · p. 10

Ethics Statement

• The study is based on 2,360 algorithmic questions from the fully open-source LeetCode platform.Source: Ethics Statement · p. 10
• All study data come from publicly available sources and involve no personal privacy information.Source: Ethics Statement · p. 10
• Data collection and processing are reported to adhere to source-platform terms and privacy policies while respecting user rights and platform regulations.Source: Ethics Statement · p. 10

A.1 Baseline Data Statistics

• Table 5 reports the size of the baseline data.Source: §A.1 · p. 12, Table 5 · p. 12
• Table 5 reports the logical sources of the synthetic data.Source: §A.1 · p. 12, Table 5 · p. 12
• The table is titled “Data Statistics.”Source: Table 5 · p. 12

A.2 Complete Main Results

• Table 6 contains the complete results for all benchmarks.Source: §A.2 · p. 12
• Table 7 gives histogram results for baseline data and LogicPro across all 27 BBH subsets.Source: §A.2 · p. 12
• Table 7 reports base-data results without adding synthetic data.Source: §A.2 · p. 12

B Details of Analysis

• Tables 8 and 9 contain complete ablation and baseline performance-gain results; Table 7 reports results using only two types of base data, without synthetic data.Source: B Details of Analysis · p. 12, Table 8 · p. 14, Table 7 · p. 14
• Table 6 gives complete benchmark evaluation results, while Figure 7 shows Qwen2-7B and Llama3-8B performance across 27 BBH subsets.Source: Table 6 · p. 13, Figure 7 · p. 13

C.1 Complete Prompts for The Three Steps in The Main Methodology

• Figures 8–11 provide the complete prompts for data collection and steps 1, 2, and 3.Source: §C.1 · p. 14, Figure 8 · p. 15, Figure 9 · p. 15

C.2 Prompts in the remaining modules

• Figures 12 and 13 provide the Problem-Program Consistency and solvability-check prompts, and Figure 14 shows the model-evaluation prompt.Source: §C.2 · p. 14, Figure 12 · p. 16, Figure 13 · p. 16

D LogicPro Examples

• Figures 15 and 16 show synthetic data examples based on the 24 Game and stair-climbing algorithm questions.Source: D LogicPro Examples · p. 14, Figure 15 · p. 18, Figure 16 · p. 19