PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing

Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen, Shafiq Joty

arXiv:2402.00658v3cs.AIcs.CL

TL;DR

LLMs can produce misleading rationales in complex reasoning, while planning-based online assessment incurs latency and cost. The paper estimates process rewards through offline simulation of trajectories and trains with DPO; it reports improvements across reasoning tasks and rationale quality, though the simulation remains resource-intensive.

  • Problem

    LLMs can generate misleading rationales, especially in complex reasoning, and online planning incurs latency and cost from frequent intermediate-state assessment and large search spaces.

    Source: §1 · pp. 1–2

  • Method

    The paper uses offline simulation to estimate intermediate-step values, builds process and trajectory rewards, and optimizes the policy with DPO.

    Source: §3.4 · p. 5, §6 · pp. 8–9

  • Results

    The authors report significant improvements on logical and mathematical reasoning, as well as improved rationale quality and conciseness with less reliance on human annotations.

    Source: §1 · pp. 2–3, §6 · p. 9

  • Takeaways & Limitations

    The results support learning planning-based reasoning from synthesized process rewards and collected trajectories across the evaluated reasoning tasks.

    Source: §1 · p. 2, §6 · p. 9

    The simulation-based approach requires substantial resources, restricting analyses such as competition-level code generation and experiments with larger policy models.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated significant potential in handling complex reasoning tasks through step-by-step rationale generation. However, recent studies have raised concerns regarding the hallucination and flaws in their reasoning process. Substantial efforts are being made to improve the reliability and faithfulness of the generated rationales. Some approaches model reasoning as planning, while others focus on annotating for process supervision. Nevertheless, the planning-based search process often results in high latency due to the frequent assessment of intermediate reasoning states and the extensive exploration space. Additionally, supervising the reasoning process with human annotation is costly and challenging to scale for LLM training. To address these issues, in this paper, we propose a framework to learn planning-based reasoning through Direct Preference Optimization (DPO) on collected trajectories, which are ranked according to synthesized process rewards. Our results on challenging logical reasoning benchmarks demonstrate the effectiveness of our learning framework, showing that our 7B model can surpass the strong counterparts like GPT-3.5-Turbo.

1 Introduction

The paper targets unreliable LLM rationales and the cost of online planning and human process annotation. It proposes offline trajectory simulation to synthesize process rewards and train reasoning policies with DPO.

  • LLMs can produce misleading rationales, particularly in complex reasoning, while human process supervision costs more than outcome annotation or automatic teacher-LLM annotation.Source: §1 · pp. 1–2
  • Search-based planning can incur substantial latency and cost because it frequently assesses intermediate states across a large search space.Source: §1 · p. 2
  • The method samples intermediate states from seed trajectories, estimates their expected returns through repeated completions, and uses PRM-ranked trajectory pairs for DPO training.Correct terminal outcomes provide the supervision used to estimate process rewards.Source: §1 · p. 2
  • The paper frames its contribution as offline simulation and trajectory collection that synthesize process rewards from outcome annotations for planning-based reasoning.Source: §1 · p. 2
  • The authors report improvements on logical and mathematical reasoning over robust baselines, alongside more concise, higher-quality rationales and reduced reliance on human annotations.Source: §1 · pp. 2–3

2 Related Work

Related work addresses reasoning through richer rationales and search, as well as optimizing LLMs from sparse feedback. The paper positions its approach alongside these directions and concurrent process-reward estimation work.

2.1 LLMs for Reasoning

2.2 Improving LLMs via Sparse Feedback

3 Method

The method treats natural-language reasoning as policy-generated state-action trajectories and estimates intermediate-step value through offline simulation. A trained process reward model scores trajectories for DPO, which uses these scores to form preferences.

3.1 Formal Definition of Natural Language Reasoning

  • The reasoning task is modeled as an MDP trajectory of states and actions generated by the policy, with final-answer correctness providing the outcome reward.The policy is optimized to maximize expected rewards.Source: §3.1 · p. 3

3.2 Estimate Process Rewards via Offline Simulation

  • Offline simulation estimates an intermediate step’s value by repeatedly completing reasoning from that point and using the resulting outcome rewards.The method samples K completions from an intermediate action or state; Figure 3 places this simulation after collecting seed trajectories.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 Synthesized Process Reward Model

  • A process reward model is trained on simulated intermediate-step values to approximate rewards more efficiently and robustly than repeated heuristic simulation.The reward-model task is formulated as classification with K classes and a cross-entropy loss.Source: §3.3 · pp. 4–5

3.4 Reward Annotation and Preference Dataset Construction

  • The method accumulates intermediate-step process rewards into a trajectory-level score and uses a minimum-success threshold to guard against hallucinated rationales affecting reward estimates.The policy is optimized with DPO rather than PPO because of PPO training instability.Source: §3.4 · p. 5

3.5 Direct Preference Optimization

  • DPO preferences use process rewards to compare trajectories with correct predictions, extending outcome-only DPO, which pairs correct and incorrect solutions.The final objective combines outcome and process preference datasets.Source: §3.5 · p. 5

4 Experiments

The experiments evaluate logical reasoning on ReClor and LogiQA-v2 and mathematical reasoning on GSM8K and MATH. Comparisons include foundational LLMs, supervised fine-tuning, preference and rejection-sampling methods, and reinforcement learning.

4.1 Datasets

  • Evaluation covers the ReClor and LogiQA-v2 logical-reasoning benchmarks and the GSM8K and MATH mathematical-reasoning test sets.ReClor and LogiQA-v2 are multiple-choice question-answering datasets.Source: §4.1 · p. 5

4.2 Baselines

  • Baselines include foundational LLMs, teacher-sampled supervised fine-tuning, outcome-supervised DPO and IPO, rejection-sampling approaches, and reinforcement-learning algorithms.Named foundational models include Llama2-70B-Chat, Mixtral-MoE-8×7B-Instruct, GPT-3.5-Turbo, and GPT-4-Turbo.Source: §4.2 · pp. 5–6

4.3 Evaluation and Implementation

  • For open-ended generation, accuracy counts only solutions that satisfy the format requirements.The paper directs readers to appendices for detailed evaluation rules and implementation details.Source: §4.3 · p. 6

5 Results and Analysis

Across logical and mathematical reasoning experiments, pDPO improves on outcome-only DPO, while annotation-sparsity and rationale-quality analyses favor process supervision. Reward analyses and a case study also examine how the approach scores and refines reasoning traces.

5.1 Overall Results on Logical Reasoning

  • DPO significantly boosts SFT; pDPO adds 2.4% and 1.3% absolute on LogiQA-v2 and ReClor, respectively, plus 2.2% in-domain on ReClor, outperforming Mixtral and GPT-3.5 Turbo.LogiQA-v2 training also generalizes to ReClor better than ReClor-trained in-domain models; the authors link this to LogiQA-v2's broader range of complex reasoning abilities.Source: §5.1 · p. 6

5.2 Improvements by Iterative Training

  • All four iterative approaches improve in-domain; pDPO surpasses PPO, while GRPO outperforms PPO using group-averaged rewards to calculate advantages.The authors attribute pDPO's edge over PPO partly to synthesized reward noise; DPO-based training takes under 16 hours on four H100 GPUs, versus over 40 hours for PPO and GRPO.Source: §5.2 · pp. 6–7

5.3 Results on Mathematical Reasoning

  • On GSM8K, synthesized rewards enhance mathematical reasoning, and 2B DPO and pDPO models significantly outperform Gemma-7B-Instruct; Gemma-2B-DPO remained challenged on MATH.The authors expanded experiments to DeepSeekMath-7B-Instruct, using 55,000 MetaMath questions augmented from MATH training data.Source: §5.3 · p. 7

5.4 Reliance on Annotations of Outcome Supervision

  • With 40% outcome annotations (3,234 questions), pDPO reached 53.5 test accuracy versus DPO's 53.9, using a process-reward model trained with 10% outcome annotations.The figure compares DPO, pDPO, and SFT accuracy on validation and test sets across annotation ratios.Source: Figure 4 · p. 7, §5.4 · p. 7

5.5 Auto-evaluation of Rationale Quality by GPT-4

  • On 261 jointly correct cases, GPT-4 rated pDPO higher overall in around 67.8%, more reasonable in 52.5%, and more concise in nearly 60%.GPT-4 compared rationales for reasonableness, concision, and logical consistency.Source: §5.5 · p. 8, Figure 6 · p. 8

5.6 Analysis of Predicted Rewards

  • Raw reward logits stay relatively stable for about 15 steps before falling, while accumulated probability rewards decrease with longer reasoning traces.The authors suggest data imbalance may reduce reward-model confidence on longer traces and note that accumulated rewards could penalize extremely long solutions.Source: §5.6 · p. 8, Figure 5 · p. 8

5.7 Case Study

  • In the case study, pDPO corrects a flawed intermediate step and removes redundant content present in the vanilla DPO solution.Source: §5.7 · p. 8

6 Conclusion

The paper recasts planning-based reasoning as a learning problem, using offline simulation to estimate intermediate-step values and avoid online-search latency. It trains a process reward model from those estimates and optimizes the policy with DPO, reporting effectiveness on logical and mathematical reasoning.

6 Conclusion

  • The framework turns reasoning-as-planning into a learning problem to avoid latency from online search.Offline simulation estimates intermediate-step values; a process reward model annotates full trajectories with sequence-level rewards before policy optimization with DPO.Source: §6 · pp. 8–9
  • The method fits a process reward model to collected expected values, assigns sequence-level rewards to trajectories, and optimizes the policy with DPO.The expected values are estimated through offline simulation of intermediate reasoning steps.Source: §6 · pp. 8–9
  • Experiments on logical and mathematical reasoning demonstrate the proposed method's effectiveness.The authors identify weak-supervision-based process reward estimation as future work to further reduce reliance on human annotations and enable consistent self-improvement.Source: §6 · p. 9

Limitations

The authors report that simulation requires substantial resources, limiting some analyses. Their baseline comparisons include foundational models, fine-tuned models, preference-optimization methods, rejection sampling, and reinforcement learning.

Limitations

  • Resource demands restrict analysis, including competition-level code generation with long contexts and experiments on larger policy models.The authors attribute these restrictions to the resources required by the simulation-based approach.Source: Limitations · p. 9

A Baseline

  • Baselines include foundational LLMs, teacher-distilled SFT models, outcome-based preference optimization, and rejection-sampling approaches.The rejection-sampling comparisons include outcome-filtered variants and a variant that ranks retained solutions with the trained PRM.Source: A Baseline · p. 12
  • Iterative-training comparisons use PPO and GRPO with binary outcome rewards and PRM-derived process rewards.For RL, process rewards use unnormalized final-layer logits; pDPO instead uses PRM probabilities.Source: A Baseline · p. 12

B Evaluation Details

Evaluation uses different prompting for untuned and fine-tuned models, and accuracy is calculated only for solutions meeting format requirements. The rules exclude multiple final predictions, truncated solutions with gathered options, and incorrectly formatted summaries.

B Evaluation Details

  • Untuned models receive a one-shot ReAct prompt, while fine-tuned models are evaluated without the demonstration; GPT-4-Turbo is tested on only 250 LogiQA-v2 questions.The one-shot prompt matches the one used for data collection; the GPT-4-Turbo restriction reflects budget limits.Source: B Evaluation Details · p. 12
  • Accuracy excludes solutions with multiple final predictions, truncated outputs that contain option indices, or incorrectly formatted summaries.These are among the cases treated as incorrect under the format and scoring rules.Source: B Evaluation Details · p. 12
  • DeepSeekMath outputs receive only basic cleaning, such as removing redundant newline symbols, because the model is already fine-tuned on chain-of-thought solutions.Source: B Evaluation Details · p. 12

C.1 Data Preparation

Data preparation uses teacher-generated responses to fine-tune models and collect solutions, with sampling and filtering choices varying by dataset. The paper also specifies dataset fractions for PRM training and task-specific hyperparameters.

C.1 Data Preparation

  • Preference-pair construction uses training-set questions and solutions sampled from the corresponding SFT model, with teacher responses used for dataset-specific preparation.Teachers generate ten solutions per question at temperature 0.7; ReClor solutions with fewer than eight reasoning steps are removed.Source: §C.1 · pp. 12–13, Table 3 · p. 13

C.1.1 Training Data Collection For PRM

  • PRM training uses 10% of LogiQA-v2 questions, 20% of ReClor, 25% of Gemma-2B data, and around 10% of DeepSeekMath data.Source: §C.1.1 · p. 13

C.2 Hyper-Parameters

  • Hyperparameters differ by task: β and C are 0.1 and 2 for logical reasoning, versus 0.5 and 3 for mathematical reasoning.σ is 0.4 for ReClor, 0.5 for LogiQA-v2 and Gemma-2B, and 0.3 for DeepSeekMath.Source: §C.2 · p. 13

C.3 Training

The comparison with MATH-Shepherd distinguishes this paper’s focus on logical reasoning and DPO training from verification- and PPO-based approaches. The appendix also describes training and evaluation procedures.

C.3 Training

  • Experiments use NVIDIA A100 and H100 GPUs with vLLM inference; Table 4 reports LogiQA-v2 accuracy by σ and defines P. Pairs as process-supervised sample pairs.The evaluation passage names greedy decoding for logical reasoning.Source: §C.3 · p. 13, Table 4 · p. 13
  • Logical-reasoning checkpoints are selected by development-set performance, and non-RL experiments are repeated with three random seeds and averaged.Gemma-2B checkpoint selection uses GSM8K, while DeepSeekMath uses MATH; RL-based approaches are run once.Source: §C.3 · p. 13

D Compared with MATH-Shepherd

  • Compared with MATH-Shepherd, this work uses PRM-guided preference data for DPO on logical reasoning, a setup that often uses fewer resources and learns more stably than GRPO.The authors say their approach does not really surpass GRPO.Source: D Compared with MATH-Shepherd · p. 13

E Effect of Different Reward Margins

The reward margin σ trades preference-pair quality against dataset size: stricter margins yield fewer pairs, while looser margins admit more false positives.

E Effect of Different Reward Margins

  • Higher σ can raise the true-positive-pair ratio but reduces training data and the chance of including hard negatives; lower σ admits more hard negatives and false positives.The passage frames σ as controlling the confidence interval between sample pairs that both reach the correct answer.Source: E Effect of Different Reward Margins · pp. 13–14
  • σ = 0.7 adds only 10% extra preference pairs and improves less significantly than σ = 0.5, while σ = 0.3 adds over 25% but performs worse than vanilla DPO.Vanilla DPO uses outcome-based preference pairs only.Source: E Effect of Different Reward Margins · p. 14
  • For zero-shot ReAct sampling, the exemplar is removed while the rest of the prompt remains unchanged.Source: Figure 7 · p. 15
  • The response-comparison prompt asks evaluators to judge reasoning for reasonableness, concision, and logical consistency.Figure 8 captions a prompt for comparing two responses across different aspects.Source: E Effect of Different Reward Margins · p. 16, Figure 8 · p. 16
  • Figure 9 presents a case study comparing LogiQA-v2 responses from Llama2-7b-DPO and Llama2-7b-pDPO.Source: Figure 9 · p. 17, E Effect of Different Reward Margins · p. 17
Loading 2402.00658v3…