PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Consistency Preference Optimization

Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, Jane Yu

arXiv:2411.04109v3cs.CLcs.AIcs.LG

TL;DR

Existing self-training methods struggle on complex reasoning because assigning correctness rewards without human annotation is difficult. SCPO instead uses answer consistency across sampled model responses to form weighted preferences during iterative self-training. It improves GSM8K, MATH, and ZebraLogic performance, approaching supervised results and outperforming several larger models on ZebraLogic.

  • Problem

    Self-training reasoning models without gold labels remains difficult because model self-evaluation can fail to judge correctness on complex problems.

  • Method

    SCPO iteratively trains models to prefer highly self-consistent over inconsistent responses, using vote-based weighted preference pairs from unlabeled and optionally labeled data.

  • Results

    SCPO improves reasoning accuracy across GSM8K, MATH, and ZebraLogic, comes within < 1% of supervised performance on GSM8K and MATH, and lets Llama-3 8B outperform larger models on ZebraLogic.

  • Takeaways & Limitations

    Self-consistency can support effective self-alignment on reasoning tasks without additional gold labels, with semi-supervised training providing further improvement.

  • Takeaways & Limitations

    SCPO-trained models may retain undesirable behavior and biases similar to the base model, motivating further evaluation and mitigation studies.

Abstract

from arXiv · show

Self-alignment, whereby models learn to improve themselves without human annotation, is a rapidly growing research area. However, existing techniques often fail to improve complex reasoning tasks due to the difficulty of assigning correct rewards. An orthogonal approach that is known to improve correctness is self-consistency, a method applied at inference time based on multiple sampling in order to find the most consistent answer. In this work, we extend the self-consistency concept to help train models. We thus introduce self-consistency preference optimization (ScPO), which iteratively trains consistent answers to be preferred over inconsistent ones on unsupervised new problems. We show ScPO leads to large improvements over conventional reward model training on reasoning tasks such as GSM8K and MATH, closing the gap with supervised training with gold answers or preferences, and that combining ScPO with standard supervised learning improves results even further. On ZebraLogic, ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and Claude-3 Haiku.

1. Introduction

SCPO addresses the difficulty of self-training reasoning models without reliable correctness rewards by using self-consistency to construct preferences from model-generated data. Across GSM8K, MATH, and ZebraLogic, it substantially improves accuracy without gold answers and approaches or exceeds supervised and larger-model baselines.

  • Motivation: Self-training methods face a bottleneck because human annotation is costly, while model self-evaluation can fail on complex tasks with unambiguous answers.SCPO is motivated by replacing unreliable self-evaluation with consistency across multiple sampled solutions.
  • Contribution: SCPO applies self-consistency during unsupervised self-training, preferring the most consistent response over the least consistent response without gold solutions or final answers.The method also weights each preference pair according to model confidence in the pair.
  • Results: 22.74% and 5.26% absolute zero-shot accuracy gains on GSM8K and MATH, respectively, brought SCPO within < 1% of the supervised baseline.These results use two iterations of unsupervised SCPO on Llama-3 8B models.
  • Results: 6.5% higher puzzle accuracy on ZebraLogic enabled SCPO-trained Llama-3 8B to outperform Llama-3 70B, Gemma-2 27B, and Claude-3 Haiku.

2. Self-consistency Preference Optimization

SCPO iteratively generates or reuses unlabeled reasoning problems, samples multiple responses, and turns answer-consistency votes into weighted preference pairs. It trains successive models to favor highly consistent responses while filtering low-consistency data and optionally incorporating gold-labeled examples.

  • Iterative Training: SCPO is an unsupervised iterative method that combines existing unlabeled queries with newly generated problems and uses self-consistency for both data generation and preference construction.The model is updated across iterations, with each trained model initializing the next one.
  • Generating New Problems: SCPO generates additional problems with few-shot prompting without requiring an accurately generated corresponding answer, allowing more diverse problem generation.Generated problems are filtered using self-consistency before training.
  • Building Self-Consistency Preference Pairs: SCPO selects the highest-vote response as the winner and the lowest-vote response as the loser, retaining pairs only when the winner exceeds threshold τ.Votes are relative frequencies of final answers among k sampled responses.
  • SCPO Loss Function: The weighted SCPO loss gives larger influence to preference pairs with larger vote margins, treating the margin as a proxy for preference quality and confidence.Its loss combines DPO and NLL terms under an unsupervised weighted objective.
  • Semi-Supervised Training with SCPO: The method can incorporate gold-labeled queries by pairing correct sampled responses with incorrect ones alongside unlabeled SCPO data.

3. Experimental Setup

SCPO is evaluated on math and logical reasoning benchmarks using Llama-3 seed models, unsupervised and semi-supervised baselines, and greedy or self-consistency inference.

  • Datasets and Metrics: GSM8K, MATH, and ZebraLogic measure mathematical and logical reasoning, with exact-answer, final-answer, puzzle, and cell-accuracy metrics.GSM8K uses 6.7K/0.8K/1.3K train/dev/test problems; MATH uses 6.7K/0.8K/5K; ZebraLogic contains 1K logic-grid puzzles.
  • Base Models: Llama-3 Base 8B initializes GSM8K and MATH experiments, while Llama-3 Instruct 8B initializes ZebraLogic experiments.The instruction-tuned model is avoided for GSM8K and MATH because it may already have been trained on their gold data.
  • Preference Training Data: SCPO uses unlabeled seed queries and model-generated problems to build consistency-based preference data across iterative training.Generated GSM8K and MATH problems need valid questions but not answers; problems with less than half consistent votes are filtered.
  • Baselines: The comparison includes gold-answer IRPO, external-reward-model IRPO, LMSI self-consistency finetuning, and the seed model with greedy or 8-way self-consistency inference.IRPOGold uses correct and incorrect gold-labeled responses, IRPORM selects responses by external reward, and LMSI uses consistency-generated targets with supervised finetuning.
  • Hyperparameters: Responses are sampled at temperature 0.7 and top-p 0.9, with k = 8 for GSM8K and MATH and k = 16 for ZebraLogic.Training uses 10 epochs, learning rate 5e-6, effective batch size 16, β = 0.5, and α = 1.

4. Main Results

SCPO improves reasoning accuracy over unsupervised baselines, approaches or exceeds gold-supervised preference training, and makes an 8B model competitive with substantially larger models on ZebraLogic.

  • Math Reasoning: SCPO outperforms unsupervised baselines: two iterations exceed LMSI by 7.20% on GSM8K, 2.76% on MATH, and 1.3% on ZebraLogic overall accuracy.On ZebraLogic, one iteration also beats IRPORM by 5.7% in puzzle accuracy and 5.5% in cell accuracy.
  • Math Reasoning: A second SCPO iteration improves greedy accuracy over the first by 2.88% unsupervised and 3.03% semi-supervised on GSM8K, and by up to 2.36% on MATH.Eight-way self-consistency accuracy saturates after the first iteration, and a third iteration provides minimal gains.
  • Math Reasoning: Two iterations of unsupervised SCPO approach gold-supervised IRPO within < 1% greedy and < 2% 8-way self-consistency accuracy on GSM8K and MATH.Vote shares correlate 0.8 with test accuracy, while generated problems help compensate for fewer confidently bootstrapped MATH examples.
  • Math Reasoning: Semi-supervised SCPO exceeds IRPOGold by 2.35% greedy and 2.19% self-consistency accuracy on GSM8K, while one iteration exceeds it by 1.24% greedy accuracy on MATH.With Llama-3.1 Base 8B, two semi-supervised iterations improve the seed model by 25.32% on GSM8K and 8.66% on MATH.
  • ZebraLogic: A Challenging Logical Reasoning Task: On ZebraLogic, two SCPO iterations improve the seed model by 6.5% puzzle accuracy and raise its leaderboard position from 38th to 30th.The resulting model is reported as the best 8B-scale LLM on ZebraLogic.
  • ZebraLogic: A Challenging Logical Reasoning Task: SCPO-trained Llama-3 8B surpasses Llama-3 Instruct 70B, Gemma-2 27B, and Claude-3 Haiku by 0.9%, 1.8%, and 3.8% in ZebraLogic puzzle accuracy.SCPO models also achieve the highest cell accuracy, with easy-puzzle gains reaching 10.3%.

5. Ablations and Analysis

SCPO becomes more consistent across iterations, while weighted preference loss and thresholding improve the quality and usefulness of self-consistency training data. Self-consistency also produces cleaner preference orderings than ArmoRM across the evaluated datasets.

  • Importance of weighted SCPO loss: Weighted SCPO loss consistently outperforms its unweighted version, improving greedy accuracy by 2.5% on GSM8K and 1.44% on MATH in the first iteration.In the second iteration, weighted models outperform unweighted counterparts by roughly 1% on both datasets.
  • Models become more consistent across iterations: SCPO increases the vote share V(y+)/k of the most consistent response across iterations and datasets.The analysis attributes this pattern to improving accuracy, additional preference-training rounds, and the training objective’s consistency pressure.
  • Models become more consistent across iterations: On ZebraLogic, the model is least consistent and accurate, whereas GSM8K is the most consistent and accurate among the analyzed tasks.
  • Impact of consistency-based filtering: A vote threshold of τ = 0.5k best balances preference-pair quality and quantity, improving downstream MATH performance by 1.92% before performance declines at τ = 0.7k.Increasing τ from 0.1k to 0.7k raises the accuracy margin from 18% to 68% while reducing data from 6.7K to fewer than 700 pairs.
  • Comparison of self-consistency to RMs: Self-consistency yields fewer incorrect pairwise orderings than ArmoRM across all three datasets and 12.3% more correct orderings on out-of-distribution ZebraLogic.Self-consistency also produces more ties, which SCPO ignores by assigning zero loss weight.

6. Related Work

Prior self-training reduces reliance on human annotation but struggles with generating or validating complex reasoning solutions. This work addresses the setting without gold solutions by using self-consistency to construct preference pairs and a consistency-weighted preference objective.

  • Prior self-training methods target iterative improvement, but LLMs struggle to evaluate their own responses when complex problems have unambiguous correct answers.
  • Human judgments remain costly for training reward models on complex reasoning tasks, leaving training without gold solutions or final answers largely unaddressed.
  • Self-consistency has improved inference-time performance across math, code generation, summarization, and question answering, while this work applies it during training for reasoning tasks.
  • SCPO differs from self-consistency fine-tuning with NLL loss by using a preference optimization loss weighted according to answer consistency.

7. Conclusion

The paper introduces SCPO, which iteratively trains models to prefer consistent over inconsistent answers without additional gold labels. Across GSM8K, MATH, and ZebraLogic, it improves reasoning performance, including outperforming larger models on ZebraLogic and improving further in semi-supervised settings.

  • SCPO iteratively optimizes models to prefer consistent answers over inconsistent ones, extending inference-time self-consistency into self-training without additional gold labels.
  • SCPO is effective across GSM8K, MATH, and ZebraLogic, outperforming several larger state-of-the-art language models on ZebraLogic.
  • Semi-supervised SCPO improves performance further when gold labels are combined with unlabeled inputs.The paper identifies extending SCPO to tasks without easily parsed final answers and evaluating more model suites and tasks as future work.

Impact Statement

SCPO trains models on math and logical reasoning without gold labels, but its outputs may retain undesirable behavior and misuse risks similar to other fine-tuned LLMs.

  • SCPO removes the need for gold labels when training LLMs on math and logical reasoning tasks.
  • SCPO-trained outputs may exhibit undesirable behavior and misuse potential similar to other fine-tuned LLMs, motivating further bias evaluation and mitigation studies.

A. Relationship between Consistency and Accuracy

Self-consistency is positively associated with answer accuracy across the evaluated reasoning datasets, although the relationship is weaker on the more difficult MATH task. Increasing the sample count affects this association only modestly while increasing computation.

  • A. Relationship between Consistency and Accuracy: Higher vote share is positively and strongly correlated with accuracy across GSM8K, MATH, and ZebraLogic.Table 6 reports this relationship using Somers’ D rank-order correlation.
  • A. Relationship between Consistency and Accuracy: The consistency–accuracy association is weakest on MATH, likely because its difficulty makes consistent answers harder to produce.
  • A. Relationship between Consistency and Accuracy: Lower sample counts such as k = 2/4 yield lower consistency–accuracy correlations because fewer answers receive multiple votes.
  • A. Relationship between Consistency and Accuracy: Using k = 16 produces slightly higher correlations, but SCPO prioritizes computational efficiency during data generation.

B. Transduction During Inference

A third SCPO training round adds less than 1% over the second, whereas bootstrapping preference pairs from test-set exemplars yields larger gains on GSM8K and MATH.

  • B. Transduction During Inference: +1.44% on GSM8K results from generating problems and preference pairs using test-split queries as exemplars instead of train-split queries.The strategy is intended to adapt better to the test set’s characteristics.
  • B. Transduction During Inference: A third SCPO round yields less than 1% improvement over the second round, indicating marginal gains after two iterations.
  • B. Transduction During Inference: +1.26% on MATH is obtained when using self-consistency accuracy in the test-query bootstrapping strategy.ZebraLogic is excluded because only test samples are available.

C. Results on Math Reasoning with Llama-3.1

With Llama-3.1 Base 8B, semi-supervised SCPO achieves the strongest overall accuracy on GSM8K and MATH, while unsupervised SCPO remains the strongest unsupervised technique.

  • C. Results on Math Reasoning with Llama-3.1: SCPOSemi-Sup. yields the overall highest accuracy on GSM8K and MATH, while SCPOUnsup. is the most performant unsupervised technique.
  • C. Results on Math Reasoning with Llama-3.1: The relative ordering among baselines remains the same with Llama-3.1 Base 8B despite higher absolute performance.
  • C. Results on Math Reasoning with Llama-3.1: Two iterations of SCPOSemi-Sup. improve greedy seed-model accuracy by 25.32% on GSM8K and 8.66% on MATH.
  • C. Results on Math Reasoning with Llama-3.1: Two iterations of SCPOUnsup. improve greedy seed-model accuracy by 21.08% on GSM8K and 7.5% on MATH.

D. Prompts

The appendix provides task-specific prompts for generating problems, rephrasing ZebraLogic puzzles, and producing step-by-step solutions in prescribed answer formats.

  • D. Prompts: ZebraLogic solution generation requests reasoning and a structured JSON solution for the puzzle.The example includes house-level assignments for names and drinks.
  • D. Prompts: Response-generation prompts request step-by-step reasoning and standardized final-answer formats for MATH and GSM8K.MATH uses a boxed final answer, while GSM8K uses an ####-delimited number.
  • D. Prompts: ZebraLogic query generation rephrases puzzles by changing specified attributes while preserving the underlying logic as closely as possible.The prompt requires a new attribute map and a rephrased puzzle, illustrated by changing names and drinks to names and foods.
  • D. Prompts: GSM8K and MATH query generation uses few-shot examples to request one solvable math word problem of similar difficulty.The prompt requires all information needed for solving and outputs only the question.
Loading 2411.04109v3…