Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning

1 Introduction

• Existing post-training relies on annotated responses, ground-truth answers, or reward-model verification, limiting scalability and broader-task generalization.The paper identifies high annotation costs for generalized reward models and potential reward hacking.Source: §1 · p. 1
• Genius trains from the policy LLM’s responses to unsupervised queries, selecting optimal responses without external auxiliary resources.The framework is presented as an initial attempt to self-improve reasoning with unsupervised general queries.Source: §1 · p. 2
• Stepwise foresight re-sampling estimates candidate-step values by rolling out future steps, then guides exploration and the creation of step-level preference pairs.Foresight scores approximate the sampling distribution used to choose the next step and re-sample for training pairs.Source: §1 · p. 2
• Because a few rollouts can bias step-value estimates and add noise to self-supervision labels, ACO penalizes inconsistency between foresight scores and step advantages.The paper reports that ACO improves training stability and performance compared with SFT and DPO.Source: §1 · p. 2
• With 25K unsupervised general queries, Genius improves average performance across diverse reasoning benchmarks by >7%.The introduction also reports that scaling on general queries consistently improves with more training steps.Source: §1 · p. 2

2.1 Preliminary

• Genius generates candidate response steps, simulates future steps to self-reward them, selects the next step, and uses the resulting responses to optimize its policy.The method derives step advantages during sampling and uses them in the optimization loss.Source: §2.1 · pp. 2–3

2.2 Exploration and Exploitation via Foresight Re-Sampling

• Foresight scores use simulated future responses, with averaged log probability of remaining steps, to estimate candidate-step values and form a sampling distribution.Temperature τ controls the sharpness of the normalized distribution.Source: §2.2 · p. 3
• Genius retains M beams for exploration, selects the highest-foresight response as positive, and resamples a negative response from the foresight distribution.The re-sampling strategy is intended to balance exploration and exploitation.Source: §2.2 · pp. 3–4

2.3 Advantage-Calibrated Optimization

• ACO calibrates negative-response self-rewards with the advantage difference, relaxing the penalty when the negative sequence has greater advantage.The hyperparameter α controls the relaxation scale.Source: §2.3 · pp. 4–5

3.1 Implementation Details

• Experiments use 25K Magpie or 32K OpenHermes2.5 queries with LLaMA3.1-8B-Instruct and Qwen2.5 3B/7B, evaluated on reasoning, general, and AIME benchmarks.Foresight sampling uses M=2, N=4, and K=4, yielding 100K and 128K training pairs for Magpie and OpenHermes2.5, respectively.Source: §3.1 · p. 5

3.2 Main Results

• RL-based self-training methods show consistent advantages over supervised fine-tuning baselines, while ground-truth responses do not improve results and self-generated responses can.Source: §3.2 · pp. 5–6
• With 25K Magpie queries, Genius raises LLaMA3.1-8B-Instruct average CoT reasoning performance by 7.43%; OpenHermes training gives >2% average advantage over strong baselines.On MATH, Genius outperforms Self-Rewarding by >4%.Source: §3.2 · p. 5, Table 1 · p. 6
• Genius maintains general-domain performance with slight improvements in most cases, including Arena-Hard gains and maintained WikiBench, MMLU, and MMLU-Pro performance.Source: §3.2 · p. 6, Table 2 · p. 7

3.3 Generalization and Adaptation

• On Qwen2.5-3B and 7B, Genius achieves the highest gains over the base models among strong baselines, though gains are not larger than on LLaMA3.1-8B-Instruct.Source: §3.3 · p. 6, Figure 4 · p. 7
• 6.67%: Genius boosts AIME 2024 performance for OpenHermes-trained LLaMA3.1-8B-Instruct and Qwen2.5-7B-Instruct.Source: §3.3 · p. 7

4.1 Ablation Studies

• The sampling ablation highlights a balance between exploration and exploitation; its variants contrast look-ahead sampling with greedy selection.The greedy w/o sampling variant uses the two highest foresight-score steps for exploration and the lowest-score step as the negative response.Source: §4.1 · p. 7, Table 3 · p. 8

4.2 Post-Training Scaling Law

• As training steps increase, Genius improves smoothly and remains far from saturation, while baseline methods appear to face scaling challenges.Because of computational limits, the reported attempts are at the 10K level, and Figure 6 shows down-sampled scaling curves.Source: §4.2 · pp. 7–8

4.1 Ablation Studies

• Ablating foresight causes a 3.17%-3.25% average performance drop, indicating its contribution to the sampling strategy.Table 3 defines the w/o foresight variant as removing look-ahead and sampling from step uncertainty alone.Source: §4.1 · p. 7, Table 3 · p. 8
• Across seven reasoning benchmarks, ACO shows average performance improvement and is described as more suitable for self-training than ROPO.Source: §4.1 · p. 7, Table 4 · p. 8

• Large-scale imitation learning on curated reasoning data has been used to build domain-specific reasoning models (Yue et al., 2024; Toshniwal et al., 2024; Xu et al., 2025b).Source: §5 · p. 8
• Earlier self-training approaches synthesize reasoning trajectories but still use explicit outcome supervision or auxiliary reward models (Sun et al., 2024b; Zelikman et al., 2022; Xu et al., 2024a; Cheng et al., 2024; Zeng et al., 2024; Jiao et al., 2024).Source: §5 · p. 8
• Prior optimization work uses supervised fine-tuning or RL-based solutions, whereas this work proposes an RL objective for robust unsupervised optimization and reports promising self-training scaling (Yue et al., 2024; Prasad et al., 2024; Guo et al., 2025).Source: §5 · p. 8

• Genius targets unsupervised enhancement of LLM reasoning and addresses response sampling, self-rewarding, and robust optimization with self-curated data.Source: §6 · pp. 8–9
• The evaluation spans seven reasoning benchmarks, seven general benchmarks, and one competition-level math task.Source: §6 · p. 9
• The scaling-law analysis points to substantial potential for further scalability.Source: §6 · p. 9

A Gradient Analysis of ACO

• When the negative trajectory has higher advantage, its calibration weight and the gradient scale decrease, reducing optimization for that training pair.The calibration relaxes the negative response’s self-reward according to its advantage.Source: A Gradient Analysis of ACO · p. 12
• DPO is a special case of ACO when α → +∞, with the same gradient scale, 1 − σ(z).Source: A Gradient Analysis of ACO · p. 12
• c-DPO applies static, equal calibration to each data pair, whereas ACO provides adaptive calibration.The paper describes c-DPO as using label smoothing to alleviate noise.Source: A Gradient Analysis of ACO · pp. 12–13
• ROPO calibrates positive and negative gradients, while ACO obtains advantage values during sampling.The paper says their gradient formulations are similar when α → +∞.Source: A Gradient Analysis of ACO · p. 13

B.1 Baselines

• SFT fine-tunes the LLM on input queries paired with labeled responses.Source: §B.1 · p. 13
• SPIN iteratively refines model-generated responses against labeled ones, while STaR bootstraps from self-constructed responses through fine-tuning.Source: §B.1 · p. 13
• CoH self-prompts for positive and negative responses and trains with DPO; Self-Rewarding has the LLM score its generated responses before DPO training.Self-Rewarding uses scores from 1–5.Source: §B.1 · p. 13
• ScPO generates multiple trajectories and labels preferences using self-consistency, modified with a clustering strategy for open-ended generation.Source: §B.1 · p. 13

B.2 Setup

• Foresight sampling uses temperature 0.6, beam size M = 2, four rollouts per beam, and K = 4 foresight steps.Sampling is supported by 32 80GB A100 GPUs and accelerated with vLLM.Source: §B.2 · p. 13
• Training uses batch size 128, learning rate 5e-7, and ACO α = 1, producing datasets of 100,000 Magpie and 128,000 OpenHermes-2.5 examples.The setup keeps four training pairs per query and uses 8 80GB A100 GPUs.Source: §B.2 · p. 13
• Evaluation uses 4-shot GSM8K, 8-shot MATH, and zero-shot settings for the other benchmarks.Inference uses the default vLLM configuration with temperature 1.0.Source: §B.2 · p. 13

C Analysis of The Training Corpus

• The visualization embeds queries with multi-qa-mpnet-base-dot-v1, projects them using t-SNE, and samples 500 queries from each dataset.Source: C Analysis of The Training Corpus · p. 13
• Figure 7 marks the general training corpus in red; the authors report its distribution is distinct from and independent of the evaluation domains.Purple marks math, yellow logic, and green other reasoning datasets.Source: C Analysis of The Training Corpus · p. 13, Figure 7 · p. 14
• Training-corpus similarity to downstream tasks is reported as notably low, with ReClor and LogiQA showing the least resemblance.The authors identify these logical reasoning benchmarks as an ideal evaluation scenario for their approach.Source: C Analysis of The Training Corpus · pp. 13–14
• Table 5 compares within-task query distances with distances between target tasks and the Magpie or OpenHermes-2.5 training corpus.The caption defines these as intra- and inter-class distances.Source: C Analysis of The Training Corpus · p. 13, Table 5 · p. 14

D More Experiments on Coding Tasks

• The coding evaluation reports performance on MBPP and LiveCodeBench.Source: D More Experiments on Coding Tasks · p. 14, Table 6 · p. 14
• The section presents coding benchmarks as a way to examine a key LLM ability beyond natural-language reasoning and understanding.Source: D More Experiments on Coding Tasks · p. 14
• Genius self-training on general data is reported to benefit LLM coding performance.Source: D More Experiments on Coding Tasks · p. 14