Source-linked AI summary
Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, Min Lin
TL;DR
CoT can overlook optimal reasoning paths, while ToT finds better alternatives at substantially higher inference cost. CPO transfers ToT’s step-level preferences into training so CoT can match or exceed ToT performance with low-latency inference. Across seven datasets and three LLMs, CPO improves base-model performance by 4.3% on average without sacrificing inference speed.
Problem
CoT may overlook optimal reasoning paths, whereas ToT’s broader search substantially increases inference complexity.
Method
CPO constructs preferred and dispreferred thoughts from ToT’s search tree at each reasoning step and trains LLMs with DPO.
Results
CPO improves base-model performance by 4.3% on average across seven datasets and achieves comparable performance to ToT without sacrificing inference speed.
Takeaways & Limitations
CPO transfers ToT’s reasoning supervision into CoT inference, substantially reducing ToT’s inference-time burden while retaining comparable or better performance.
Takeaways & Limitations
CPO still requires time-consuming ToT data generation and has only been tested on text language models, not vision-language models.
Abstract
from arXiv · showhide
The recent development of chain-of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost of significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial inference burden. This is achieved through Chain of Preference Optimization (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT using the inherent preference information in the tree-search process. Extensive experimental results show that CPO significantly improves LLM performance in solving a variety of complex problems, including question answering, fact verification, and arithmetic reasoning, demonstrating its effectiveness. Our code is available at https://github.com/sail-sg/CPO.
1 Introduction
CoT can miss optimal reasoning paths because it follows a single trajectory, whereas ToT explores alternatives but increases inference complexity. CPO uses ToT’s intermediate preferences to train CoT, improving performance while avoiding that inference burden.
- CoT’s single-path focus can overlook optimal reasoning paths and produce less deliberate answers.
- ToT generates multiple branching thoughts, evaluates them, and prunes alternatives to encourage more deliberate reasoning.
- CPO constructs preferred and dispreferred thought pairs at each reasoning step from ToT’s search tree and trains LLMs with DPO.
- CPO improves average accuracy by up to 4.3% across seven datasets using LLaMA and Mistral base models.
- CPO achieves comparable or superior performance to ToT, whose inference takes more than 50 times longer on average.
2 Related Work
Prior work improves LLM reasoning through multi-step prompting, nonlinear search, self-improvement, and tree-search methods. These approaches often increase inference latency or require external supervision, reward models, or additional policy and value models.
- Multi-step reasoning and nonlinear tree or graph structures have been used to improve generated reasoning paths.
- Tree- and graph-based reasoning methods combine thought evaluation with search algorithms but require searching during inference, increasing latency.
- Self-improvement methods generate data for fine-tuning, typically relying on external reward models or related supervision mechanisms.
- MCTS can improve LLM decoding but has high inference latency and may require labeled data plus separate policy and value models.
- CPO eliminates human annotations and additional models while using tree-search supervision to tune LLM reasoning.
3 Background
The background introduces CoT as sequential reasoning, ToT as tree search over multiple thoughts, and DPO as direct optimization from preference pairs. These concepts provide the basis for CPO’s preference-based training approach.
- Chain-of-Thought Prompting: CoT generates intermediate thoughts z1, · · ·, zn that connect an input x to a final output y.
- Tree-of-Thought Prompting: ToT models reasoning as a tree search in which a thought generator proposes several thoughts and a state evaluator assesses them.
- Tree-of-Thought Prompting: ToT retains n-best thoughts for continued search, with the final result selected by an algorithm such as BFS or DFS.
- Direct Preference Optimization: DPO directly optimizes an LLM to align with preference data by maximizing the probability ratio of preferred responses.
- Direct Preference Optimization: Preference data ranks two generated completions as a preferred winner and a dispreferred loser conditioned on the input.
- Direct Preference Optimization: The DPO objective uses a logistic function, while β controls the penalty for deviations from the reference model πref.
4 Our Method: Chain of Preference Optimization
CPO converts ToT’s step-level preferences into training supervision, using search-generated preferred and dispreferred thoughts to optimize CoT reasoning without relying on complete paths alone. The method synthesizes preference pairs through thought generation, state evaluation, and pruned search, then trains with a DPO-based objective.
- Core idea: CPO uses preferences over thoughts generated at each reasoning step rather than training only on complete reasoning paths.The approach retains information from non-optimal thoughts that prior methods often discard.
- Preference construction: The synthesis procedure generates multiple thoughts, evaluates their progress, and uses breadth-first search with pruning to retain the highest-scoring paths.The search ends when a generated thought contains “so the final answer is:”.
- State evaluation: State evaluation uses verbal justification followed by likely/impossible classification, mapping the two outcomes to scores of 10 and 1.The method averages repeated evaluations after shuffling demonstration examples to reduce randomness and bias.
- Preference construction: The preference dataset contains paired preferred and dispreferred thoughts for every step of each selected reasoning chain.Thoughts on selected paths are marked preferred, while unselected child thoughts from the same parent state are treated as dispreferred.
- Training objective: CPO optimizes each preferred–dispreferred pair with a DPO-derived objective conditioned on the preceding reasoning state.The objective is applied to the chain of preference thoughts collected during ToT-style search.
5 Experiments
Experiments evaluate CPO across question answering, fact verification, and arithmetic reasoning using multiple datasets, models, and baselines. CPO improves reasoning over CoT, matches or exceeds ToT with lower inference latency, and benefits from using broad step-level preference information.
- Setup: Experiments cover question answering, fact verification, and arithmetic reasoning across seven datasets using LLaMA-family and Mistral models.Baselines include greedy CoT, tree-search ToT, and TS-SFT.
- Overall results: CPO achieves an average improvement of 4.3% and a maximum improvement of 9.7% over CoT across tasks and language models.These gains require no additional human-annotated data.
- Overall results: CPO maintains CoT’s low inference latency while delivering comparable or superior performance to ToT, averaging 57.5× faster than ToT.CPO shifts the computational burden from inference to training.
- Overall results: CPO improves over TS-SFT by 2.7% on average and reaches a maximum increase of 10.3%.The comparison attributes this advantage to using both selected and unselected thoughts, whereas TS-SFT uses only selected paths.
- Component-wise evaluations: The three dispreferred-thought selection strategies show minimal performance differences, so the study chooses CPO w/ All to obtain more preferred-thought pairs.The selected reasoning path determines preference more strongly than intermediate evaluation scores.
- Component-wise evaluations: Training on fewer than 80 instances can underperform no training, while optimization with 120 instances surpasses the base model.The authors associate the initial degradation with overfitting and report improvement as the training set grows to 120 instances.
- Component-wise evaluations: CPO improves performance by 3.2% with both uniform QA data and mixed-type data configurations.The result supports performance gains across diverse training-data mixtures.
6 Analysis
The analysis finds that incorporating dispreferred thoughts improves CPO and that per-step preference construction avoids a gradient-cancellation problem affecting full-path optimization.
- Increasing dispreferred-data inclusion consistently improves model performance, highlighting the value of using both preferred and dispreferred thoughts.
- FPO constructs preferences from complete paths, whereas CPO constructs paired preferences at each reasoning step.
- In FPO, gradients for longest common prefix tokens cancel, leaving optimization only where paired paths diverge.
- The Bamboogle dataset’s longest common prefix accounts for 28% of total length, illustrating the potential extent of FPO’s cancellation issue.
- Switching from CPO to FPO decreases relative performance by 4.6%, performing worse than baseline SFT.
7 Conclusion
The paper presents CPO as a way to use tree-of-thought supervision to improve LLM reasoning without sacrificing inference speed. It reports consistent gains across models and datasets while approaching ToT performance at much lower inference cost.
- CPO leverages supervision generated by tree-of-thought self-reasoning to enhance the reasoning ability of LLMs.
- Across three LLMs and seven datasets, CPO improves base-model performance by 4.3% on average without sacrificing inference speed.
- CPO substantially outperforms TS-SFT and achieves comparable performance to ToT, which requires approximately 57.5 times more inference time.
- Future work will integrate CPO with Graph-of-Thoughts and AlphaZero-like tree search and explore weak-to-strong alignment.
Societal Impacts and Limitations
The paper notes that CPO needs time-intensive ToT data generation, has been tested only on text models and a small set of downstream tasks, and raises misuse concerns.
- CPO still requires time-consuming data generation through ToT, motivating faster generation methods such as speculative decoding and KV-cache pruning.
- The evaluation covers only text language models, while application scope remains restricted to a small set of downstream tasks.
- The authors identify potential misuse and state that ethical considerations are necessary because misuse could cause unintended consequences.
A Detailed Experiment Configurations
The experiments limit dataset sampling to control ToT’s computational cost while using relatively few training instances to construct preference pairs without ground-truth labels.
- Each dataset uses at most 300 randomly sampled test samples, or all available samples when fewer than 300 exist.
- Training preference pairs are constructed from fewer than 300 randomly selected instances per dataset without using ground-truth labels.
- Approximately 200 samples generate about 6,531 preference pairs on average, supporting a small-sample design.
- The 300-sample choice is presented as a trade-off between efficiency and effectiveness because preference-data construction is time-intensive.
B Additional Experiments
Additional experiments show that iterative CPO improves CoT performance, while further fine-tuning can reduce ToT performance as output diversity narrows. Across comparisons, metrics, ablations, and examples, CPO remains effective and its preferred reasoning paths align more closely with ToT than CoT.
- Iterative learning: CoT performance improves as iterative training proceeds, with CPO-only performance increasing by 4% after two iterations.In contrast, ToT performance can decline after fine-tuning; in SFT+CPO, it decreased by 2.7% after the first SFT round.
- Baseline comparisons: CPO surpasses ReST and self-rewarding on average under a fair comparison using the LLM as the reward model or annotator.The comparison adapts the baselines to avoid relying on their original external reward models or labeled data.
- Additional metrics: F1 performance on the three QA tasks aligns well with the corresponding accuracy results.The additional evaluation reports F1 scores for the QA datasets.
- Ablation analysis: Ablation trends remain generally consistent across different models and datasets.The analyses cover the number of instances, dispreferred thoughts, and component-wise effects.
- Reasoning-path analysis: CPO-favored reasoning paths align more closely with ToT-selected paths than with CoT paths, indicating higher reasoning quality in the examples.Table 6 marks similar ToT and CPO paths for comparison.