Weak-to-Strong Reasoning

1 Introduction

• The paper studies the weak-to-strong setting, where supervision comes from a less capable model and a stronger model’s capabilities remain not fully elicited.It frames this setting as relevant when models exceed human capabilities and accurate, comprehensive supervision becomes difficult.Source: §1 · p. 2
• Existing methods either leave a gap to strong-model capabilities or rely on confidence loss that does not naturally extend to open-ended reasoning.The paper identifies a need to reduce overfitting to weak-supervisor errors while eliciting the strong model’s latent reasoning abilities.Source: §1 · p. 2
• The framework first selectively curates a small dataset for supervised fine-tuning, then uses strong-model-generated contrastive samples for preference optimization.The curation combines weak-model data with data generated by the strong model through in-context learning.Source: §1 · p. 2
• The method outperforms full weak fine-tuning on GSM8K and reports surpassing the gold-fine-tuned strong-model ceiling on level 4–5 MATH problems.On GSM8K with Gemma-2b supervision, the first stage improves performance by 26.99 points, followed by an additional 8.49 points through preference optimization.Source: §1 · p. 3
• On OlympicArena, Llama3-8b-instruct effectively supervises Llama3-70b, and the two-stage approach beats full weak fine-tuning by 3.19 points.The dataset has no definitive ground-truth answers.Source: §1 · p. 3

2.1 Typical Learning Paradigms for LLMs

• Generic-supervised learning uses sufficient ground-truth answers, while weak-to-strong learning uses a weaker supervisor to elicit latent knowledge from a stronger model.The paper contrasts these with distillation from a stronger teacher and self-improvement based on self-generated responses, which still relies on ground truth to assess usability.Source: §2.1 · pp. 3–4
• Semi-supervised learning reduces labeling demands but still requires a small, high-quality seed set of standard answers.The cited examples include learning from differences between self-generated and expert-annotated responses and easy-to-hard generalization from human annotations.Source: §2.1 · p. 3

2.2 Weak-to-Strong Reasoning Setup

• In the weak-to-strong reasoning setup, the strong model receives questions without reasoning chains or answers, while a weaker model is trained on a separate gold-solution split to supervise it.The original training set is divided into two equal parts: the weak model uses Dgold,1, while the strong model accesses questions from Dgold,2.Source: §2.2 · p. 4

3 Methodology

• Stage I uses selected positive samples for supervised fine-tuning; Stage II applies preference learning to potentially negative weak data.The framework is designed to maximize use of weak data while eliciting the strong model’s capabilities.Source: §3 · p. 4

3.1.1 Full Weak Fine-Tuning

• Full weak fine-tuning trains the strong model on the entire weak dataset, an approach whose effectiveness for reasoning tasks the authors investigate.Prior work validated the approach for text classification, while its efficacy in reasoning remained unexplored.Source: §3.1.1 · p. 4

3.1.2 Weak In-Context Learning

• Weak in-context learning uses four randomly selected weak examples as prompt demonstrations, whose correctness cannot be verified without ground truth..Source: §3.1.2 · p. 4

3.1.3 Weak-ICL Fine-Tuning

• Weak-ICL fine-tuning combines weak-model data with strong-model in-context solutions, retaining examples when their final answers agree.The method calls this selection rule final answer consistency and uses the resulting datasets for supervised fine-tuning.Source: §3.1.3 · p. 5
• Iterative training bootstraps performance by using models trained on distinct weak-generated and strong-model-generated data sources for another round.The authors note that the sources retain different data representations after the initial fine-tuning.Source: §3.1.3 · p. 5
• The authors report further improvement beyond ICL from their method, while noting that ICL performance depends partly on demonstration selection.Experiments in Section 4.3 are cited for this finding.Source: §3.1.3 · p. 5
• Iterative training is optional and may reduce performance when data quality is too low or the model overfits.Source: §3.1.3 · p. 6

3.2 Stage II: Learn from “Negative” Samples

• Stage II samples responses from Mplus, discards questions below a confidence threshold, and pairs chosen and rejected responses using the confident answer and weak-model response.When the weak answer matches Mplus’s confident answer, the weak response is chosen; otherwise it is rejected.Source: §3.2 · p. 6

4.1 Datasets

• GSM8K and MATH provide the evaluation data; MATH contains harder competition problems, and the training set is divided into Dgold,1 and Dgold,2.The weak model uses Dgold,1 to develop initial mathematical skills, while the strong model accesses Dgold,2 questions without ground-truth answers.Source: §4.1 · p. 6, Table 4 · p. 6

4.2 Experiment Settings

• The experiments use Llama2-70b with three weak models, apply LoRA to the strong model, and test DPO and ORPO for preference learning.GSM8K receives two Stage I iterations and MATH one.Source: §4.2 · p. 7

4.3 Results of Stage I

• On GSM8K, weak-ICL fine-tuning raises performance from 25.17 to 60.12 with Gemma-2b supervision, exceeding naive full weak fine-tuning by 31.08 points.MATH full weak fine-tuning with Gemma-2b scores below the weak floor (10.0 versus 11.6), while weak-ICL fine-tuning bridges the gap.Source: §4.3 · pp. 7–8, Figure 4 · p. 7
• Carefully selected demonstrations raise Gemma-2b weak-ICL accuracy from 56.48 to 64.06, and strong-model fine-tuning on selectively curated data reaches 64.75.The authors describe effective demonstration selection as non-trivial and beyond this paper's scope.Source: §4.3 · p. 8

4.4 Results of Stage II; 4.6 Experiments Closer to Future Scenarios

• Preference optimization improves GSM8K accuracy by 8.49 points and MATH accuracy by at least 1 point; on OlympicArena it beats full weak fine-tuning by 3.19 points.The OlympicArena model uses 746 training examples for its weak-ICL fine-tuning stage.Source: Table 5 · p. 8, §4.4 · p. 8, §4.6 · p. 9

4.4 Results of Stage II

• The Stage II preference data construction uses weak-model responses as one chosen or rejected response; using only strong-model-generated responses slightly degrades DPO accuracy.The self-generated GSM8K setting with Llama2-7b scores 62.40, a 0.22-point decrease.Source: §4.4 · p. 8

4.5 Analysis

• On harder MATH problems at levels 4–5, ORPO-enhanced Mpro surpasses the strong ceiling from supervised fine-tuning on gold solutions.Llama2-70b exceeds 30 accuracy points on level 1 after training, although Llama2-7b reaches only 6.98.Source: §4.5 · p. 9, Figure 6 · p. 9

4.6 Experiments Closer to Future Scenarios

• In the OlympicArena setting, weak-ICL fine-tuning beats full weak fine-tuning using 746 examples, and preference optimization further increases the advantage.The evaluation uses Llama3-8b-instruct as the weak model and Llama3-70b as the strong model.Source: §4.6 · p. 9, Table 6 · p. 9

5.1 LLM Training

• Supervised fine-tuning depends heavily on training-data quality, while preference methods such as DPO and ORPO use positive and negative feedback.The paper explores DPO and ORPO with contrastive samples constructed in a weak-to-strong setting.Source: §5.1 · pp. 9–10

5.2 Mathematical Reasoning

• Mathematical reasoning work includes prompting to elicit pretrained capabilities and fine-tuning on larger or higher-quality question-answer collections.Prior studies also use supervised fine-tuning and preference optimization to improve mathematical problem solving.Source: §5.2 · p. 10

• The method has the strong model autonomously refine training data and iteratively expand its learning scope without human annotations or a more advanced model.The conclusion presents self-directed curation as a route toward scaling AI capability enhancement.Source: §6 · p. 10

• Llama2-70b and Llama3-70b are proxies rather than genuine future advanced models, and process-level supervision remains unexplored.Early experiments found only limited improvements from non-negligible process-level errors and noise.Source: Limitations · p. 10

A.1.1 Dataset Construction

• GSM8K’s 7,473 original training samples are divided evenly into two subsets and supplemented to 7,000 samples each.The weak model receives Dgold,1 with questions and gold solutions; the strong model receives 7,000 questions from Dgold,2 without ground-truth answers.Source: §A.1.1 · p. 16
• GSM8K has 1,319 test samples.Source: §A.1.1 · p. 16
• For MATH, the test set is the same 500 representative problems used by Lightman et al. (2023).The remaining 12,000 samples are split evenly into Dgold,1 and Dgold,2, with 6,000 each.Source: §A.1.1 · p. 16

A.1.2 Statistics of MATH test set

• The MATH test set contains 500 samples.Source: §A.1.2 · p. 16
• Table 7 lists the difficulty-level distribution across the 500 test samples.Source: §A.1.2 · p. 16

A.2 Training Details

• Stage I uses LoRA; Stage II applies DPO or ORPO preference optimization, with one training epoch for each method.Stage I runs two GSM8K iterations and one MATH iteration; OlympicArena uses the GSM8K hyperparameters.Source: §A.2 · p. 16

A.3.1 Diversity Analysis

• Mhybrid-ft produces more varied responses than Micl-ft, while Micl-ft repeats nearly the same sampled responses for each question in over 36% of instances.The authors connect response diversity to higher pass@k and to distinguishing positive from negative samples for Stage II.Source: §A.3.1 · pp. 16–17

A.3.2 Training Accuracy of Stage I

• Stage I data show differences in process-level accuracy despite generally consistent final-answer accuracy, and low MATH accuracy explains using one iteration.Process-level accuracy requires no errors throughout the intermediate reasoning steps; the authors suggest self-generated solutions may align better with the model’s inherent distribution.Source: §A.3.2 · p. 17

A.4.1 Details of Stage I on MATH

• For MATH, consistency-filtered Stage I data were too limited for effective fine-tuning, so additional inconsistent examples were sampled.The data were supplemented to 1,000 instances for Gemma-2b and 2,000 for Mistral-7b.Source: §A.4.1 · p. 18

A.4.2 Pass@k Results

• Mhybrid-ft achieves pass@k margins of up to 5.23 points over its counterparts, except on MATH with Llama2-7b, likely because of limited training data.The paper uses the final Mhybrid-ft iteration for Stage II preference optimization, which it says converts sampling potential into greedy-decoding gains.Source: §A.4.2 · p. 18

A.4.3 PGR of Stage I

• Stage I’s PGR comparisons favor the proposed weak-ICL fine-tuning over full weak fine-tuning, but variation across weak models is not meaningful.The main experiments report test accuracy instead for a more detailed depiction of model performance.Source: §A.4.3 · pp. 18–19

A.4.4 Effect of SFT Data

• Reducing training-data quantity while improving quality can raise accuracy by over 17 points, and the method remains effective when ground truth is unavailable.The reported method is slightly below the gold-data results; those gold-filtered settings are used only for analysis, not the weak-to-strong setting.Source: §A.4.4 · p. 19

A.4.5 Zero-Shot Results

• Llama2-70b’s zero-shot results on GSM8K and MATH are significantly below its weak-ICL results.Zero-shot evaluation uses a two-stage prompt to elicit a reasoning path and then extract the answer.Source: §A.4.5 · p. 19, Table 14 · p. 19