PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer

Minju Gwak, Alan Ritter

arXiv:2610.41296v1cs.AI

TL;DR

Answer-level feedback cannot distinguish equally correct reasoning traces, so the paper tests whether confidence- and entropy-based preferences can improve learning under limited supervision. Across online and offline self-improvement settings, UID-Conj. improves aggregate performance in several comparisons, though benefits depend on the task and training setting.

  • Problem

    Answer-level feedback cannot distinguish equally correct traces that may differ in reasoning quality or value as training examples, motivating tests of auxiliary preferences without new problems or step-level annotations.

    Source: §1 · p. 1

  • Method

    The study compares predictive-confidence and entropy-trajectory preferences in online TTRL and offline training on correct responses, including DPO and weighted SFT.

    Source: §1 · p. 1, §4.3 · p. 6

  • Results

    2.08 percentage points: UID-Conj. improves the online four-benchmark macro-average over unmodified TTRL, while offline weighted SFT improves macro-average by 1.93 points over uniform SFT.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    Auxiliary preferences can extract training value from existing correct responses, and their evaluation should consider downstream learning outcomes alongside judged trace quality.

    Source: §7 · p. 9

    The experiments focus mainly on Qwen3-1.7B under limited mathematical supervision, and transfer tests do not measure unrestricted open-ended reasoning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Answer-level rewards cannot distinguish reasoning traces with equally correct outcomes. We study whether auxiliary trajectory preferences improve small-language-model self-improvement and transfer under limited supervision. We compare confidence-based (SC) and entropy-based metrics (UID-Conj.), which combines global entropy variation and local smoothness, through online reward augmentation and offline training on correct responses. Online experiments adapt Qwen3-1.7B on 30 AIME problems without reference answers; offline experiments use historical AIME problems. UID-Conj. improves the online macro-average across the adaptation benchmark and three transfer benchmarks by 2.08 percentage points over unmodified TTRL. The strongest evidence comes from offline weighted SFT: UID-Conj. exceeds uniform weighting on the same response pool by 1.93 points, with a paired prompt-bootstrap 95\% CI of [0.44, 3.44]. Benefits are task-dependent: SC rewards judged trace quality and instruction following, while UID-Conj. rewards yield stronger training outcomes in several settings. We find that auxiliary preferences can expose learning value among equally correct traces not captured during inference time, motivating their evaluation through downstream gains.

1 Introduction

The introduction asks whether preferences among equally correct reasoning traces can improve learning and transfer beyond what answer-level feedback can distinguish. It tests confidence- and entropy-based preferences in limited-supervision online and offline settings.

1 Introduction

  • Answer-level feedback cannot distinguish equally correct responses that may differ in reasoning soundness, justification completeness, redundancy, or value as training examples.The study asks whether auxiliary trajectory preferences can improve learning and downstream transfer without requiring new problems or step-level annotations.Source: §1 · p. 1
  • SC favors greater predictive confidence, while UID-Conj. combines global entropy variation with local smoothness to define preferences among responses with the same answer-level reward.UID-Conj. is inspired by uniform information density and observations about information flow in successful reasoning.Source: §1 · p. 1
  • The online test adapts with TTRL for two epochs on 30 AIME2026 problems without reference answers, augmenting majority-agreement rewards among responses supporting the majority answer.The auxiliary preferences leave the reward distinction between agreeing and disagreeing responses intact.Source: §1 · p. 2
  • 2.08 percentage points: UID-Conj. raises Qwen3-1.7B’s four-benchmark macro-average over unmodified TTRL after adaptation on 30 AIME2026 problems without reference answers.The introduction also reports gains for SC and offline preference-training comparisons.Source: §1 · p. 2
  • The study combines an independent assessment of trace quality with controlled online and offline self-improvement experiments and transfer evaluation to critique and research-level reasoning.The controlled comparisons use controls matched within each training objective.Source: §1 · p. 2

2 Related Work

Related work situates auxiliary trajectory preferences among answer selection, self-improvement feedback, and transfer beyond verifiable tasks. It identifies whether entropy-trajectory structure adds useful supervision beyond confidence as an open question.

2 Related Work

  • Self-consistency aggregates final-answer agreement, while SC scores predictive distributions (Kang et al., 2025); UID work relates successful trajectories to local smoothness and global non-uniformity.Whether these scores distinguish useful already-correct responses, and whether reinforcing them improves generation, remains unresolved.Source: §2 · p. 2
  • Self-improvement spans learning from correct rationales, outcome feedback (Shao et al., 2024; Guo et al., 2025), process supervision (Lightman et al., 2023), and response-length preferences.Related approaches use SC as a GRPO reward or minimize entropy during training.Source: §2 · p. 2
  • Math supervision offers accessible known-answer feedback, but transfer to harder mathematics and scientific or open research problems extends beyond that setting (Shao et al., 2024; Rein et al., 2023).Open research problems may lack gold answers for outcome-based rewards even when proposed solutions can be checked.Source: §2 · p. 2

3 Characterizing Trajectory Preferences: Do Auxiliary Signals Add Value beyond the Final Answer?

The section tests whether auxiliary signals add value beyond answer-level agreement, first in answer selection and then in quality comparisons among correct traces. Results show complementary but context-dependent selection and trace-quality patterns.

3 Characterizing Trajectory Preferences: Do Auxiliary Signals Add Value beyond the Final Answer?

  • The study compares answer selection and reasoning properties among verifier-correct responses, assessing selected traces with a blinded external judge.The diagnostic holds final-answer correctness fixed and examines validity, sufficiency, consistency, and relevance/nonredundancy.Source: §3 · pp. 2–3, §3.3 · p. 4

3.1 Experiment Setup

  • Selectors are evaluated on Qwen3-1.7B and Qwen3-4B using identical nested rollout pools of n ∈{2, 4, 8} from AIME2026, HMMT2026, and MinervaMath.Mean accuracy is reported over prompts, with missing selections counted as incorrect.Source: §3.1 · p. 3

3.2 Auxiliary Rewards with Answer-level Aggregation

  • 27.0%: UID-Conj. + Borda achieves the highest Qwen3-1.7B accuracy, while SC + Borda leads for Qwen3-4B at 34.7%.The gains vary by rollout budget and model size; auxiliary scores complement answer-level agreement rather than improving selection uniformly.Source: §3.2 · p. 4

3.3 Independent Assessment of Trace Quality

  • 13.2 and 11.7 percentage points: SC’s net win rates over random correct selection for Qwen3-1.7B and Qwen3-4B contrast with UID-Conj.’s −7.7 and −0.5 points.More than half of outcomes are ties, and the net win rates indicate that trace-quality benefits depend on the scoring signal.Source: §3.3 · p. 4
  • SC scores higher on substantive reasoning criteria including validity and consistency, while UID-Conj. scores higher on relevance/non-redundancy; SC’s total-score advantage is 0.584 and 0.338 points.The respective total-score advantages are for Qwen3-1.7B and Qwen3-4B, motivating tests of whether trace-selection differences translate into training effectiveness.Source: §3.3 · p. 4

4 Training with Auxiliary Trajectory Signals

The paper scores reasoning trajectories from predictive-token statistics and applies those preferences in online majority-gated RL or offline training on verified-correct responses.

4.1 Auxiliary Trajectory Scoring

  • SC summarizes predictive confidence, while UID-Conj. combines global entropy variation and local smoothness to score responses from token-level predictive statistics.The scoring framework supports online and offline training, with scoring policies, features, and normalization varying by setting.Source: §4.1 · p. 5

4.2 Majority-Gated Online RL

  • Online, majority agreement supplies a pseudo-label without reference answers; auxiliary rewards adjust only majority-consistent responses while preserving reward ordering over dissenters.Majority agreement is not a guarantee of correctness, and groups with fewer than two majority-consistent responses receive no auxiliary adjustment.Source: §4.2 · p. 5

4.3 Offline Externsions: Learning from Verified Traces

  • Offline DPO prefers the highest- over the lowest-scoring eligible correct response, with randomly selected correct-response pairs as a control.Pairs require a minimum score gap and use the common set of qualifying prompts across scoring methods.Source: §4.3 · p. 6
  • Auxiliary-weighted SFT emphasizes higher-scoring eligible correct responses, while uniform SFT controls on the same examples with identical response-level loss normalization.Weights have mean one within each prompt, preserving that prompt’s total example weight.Source: §4.3 · p. 6

5 Experiment

Across online and offline experiments, auxiliary preferences improve several benchmark aggregates, while results vary by model, training objective, and transfer task.

5.1 Experiment Setup

  • Experiments use Qwen3-1.7B and Qwen3-4B, with two online RL epochs on AIME2026 and one offline epoch on historical AIME problems.Evaluation includes competition math, GPQA-D, CriticBench, LabBench2, and instruction-following tasks.Source: §5.1 · p. 6

5.2 Results

  • 2.08 percentage points: UID-Conj. raises Qwen3-1.7B macro-average accuracy over unmodified TTRL, from 25.51% to 27.59%.SC reaches 27.54%, and both auxiliary methods outperform unmodified TTRL on all four benchmarks.Source: Table 3 · p. 7, §5.2 · p. 7
  • On Qwen3-4B, UID-Conj. preserves base AIME2026 accuracy while improving HMMT2026 and MinervaMath to 13.94% and 31.40%; TTRL raises AIME2026 from 55.33% to 60.00% but reduces out-of-domain performance.The table reports mean accuracy across competition-level math benchmarks.Source: Table 3 · p. 7, §5.2 · p. 7
  • 1.93 percentage points: UID-Conj. weighted SFT exceeds uniform SFT in macro-average accuracy, reaching 29.49%.This comparison uses the same correct response pool; the table reports offline effectiveness on competition-level math benchmarks.Source: Table 4 · p. 7, §5.2 · p. 8
  • UID-Conj. leads CriticBench verdict accuracy and incorrect-class F1 at 68.00% and 63.23%, while SC has the higher instruction-following aggregate, 39.16% versus 38.51%.The transfer table covers critique, scientific reasoning, and instruction following.Source: Table 5 · p. 8, §5.2 · p. 8

6 Further Analyses

Uncertainty intervals differ across comparisons, and an additional training dataset provides a further test of whether auxiliary-reward gains extend beyond the original setting.

Uncertainty Analysis

  • Online UID-Conj. comparisons with TTRL and Random-TTRL and the offline DPO comparison have intervals including zero; weighted-SFT and CriticBench accuracy/F1 comparisons against TTRL have positive intervals.The estimates use paired prompt-cluster bootstrap confidence intervals.Source: §6 · p. 8
  • CriticBench comparisons against Random-TTRL and the LabBench2 comparison against TTRL have intervals that include zero.These comparisons therefore do not have positive intervals in the reported analysis.Source: §6 · pp. 8–9

Training with a different dataset

  • With S1K training, both SC and UID-Conj. improve over standard TTRL on most benchmarks, and UID-Conj. outperforms TTRL across five.The analysis evaluates four verifiable reasoning benchmarks plus CriticBench and LabBench2.Source: §6 · p. 9, Table 7 · p. 9

7 Conclusion

Auxiliary trajectory preferences can extract training value from correct responses that answer-level feedback alone does not distinguish. The results separate judged trace quality from training utility and motivate evaluating preferences by downstream learning outcomes.

7 Conclusion

  • Correct answers do not exhaust the learning signal in model-generated reasoning.Auxiliary trajectory preferences can extract additional training value from existing responses under limited supervision.Source: §7 · p. 9
  • SC selects higher-rated traces, while UID-Conj. yields larger gains in offline training comparisons and selected transfer tasks.Source: §7 · p. 9
  • Trajectory preferences should be evaluated by the learning outcomes they support alongside their ability to identify high-quality reasoning.Source: §7 · p. 9

Limitations

The evaluation is limited to particular models, supervision, trace-quality judgments, and transfer tasks. The online comparisons also do not fully isolate trajectory-specific preferences from reward variation in unanimous groups.

Limitations

  • Online comparisons do not fully distinguish trajectory-specific preference benefits from reward variation in unanimous groups.The authors use a control that permutes auxiliary terms within majority-agreeing groups to more closely isolate their association with individual trajectories.Source: Limitations · p. 9
  • The trace-quality diagnostic uses one model judge and only verifier-correct responses.Source: Limitations · p. 9
  • Transfer evaluation covers critique-verdict accuracy, evidence-conditioned scientific responses, and programmatically verified instruction following, not unrestricted open-ended reasoning.Broader response cohorts and human validation would extend the assessment.Source: Limitations · p. 9

Broader Impact and Ethical Implications

Auxiliary signals may help make reasoning improvement more accessible where larger models or extensive supervision are impractical, but trajectory generation and scoring incur computational costs. Self-improvement can reinforce errors, so observed benchmark gains do not establish dependable autonomous scientific judgment.

Broader Impact and Ethical Implications

  • Improving compact models from limited training data may make reasoning systems more accessible where larger models or extensive supervision are impractical.Source: Broader Impact and Ethical Implications · p. 9
  • Generating and scoring multiple trajectories incurs computational costs, and the experiments do not establish net energy or cost savings.Source: Broader Impact and Ethical Implications · p. 9
  • Majority agreement and smooth entropy trajectories do not guarantee factual or logical validity, and gains do not establish dependable autonomous scientific judgment.The authors recommend independent verification where available and expert oversight when incorrect outputs could have consequential effects.Source: Broader Impact and Ethical Implications · p. 9

A.1 Selector Metric Calculation

The section defines confidence- and entropy-based trajectory scores, then describes how they select or weight responses and how auxiliary rewards affect online and offline training. It also details a correct-trace quality diagnostic and the conditions under which auxiliary rewards can produce learning signals.

A.1 Selector Metric Calculation

  • UID-Conj. combines global entropy variation with local smoothness, while Self-Certainty (SC) ranks responses by predictive confidence; selector rules use calibrated scores to choose among eligible responses.UID features use token entropies in relative-position bins; higher global variation and smoother local changes correspond to larger feature values.Source: §A.1 · p. 12

A.2 Details of Trace Quality Assessment

  • Correct-trace comparisons use blinded, independent rubric scores for validity, sufficiency, consistency, and relevance, with a common cohort requiring all contrasted responses to be verifier-correct.The analysis is conditional on saved response pools and is not exclusively held out; it does not establish cross-judge agreement or training-reward benefits.Source: §A.2 · pp. 12–13

B.1 Auxiliary Trajectory Scoring

  • Auxiliary trajectory scoring maps token-level confidence or entropy features to calibrated empirical midranks; UID-Conj. multiplies global-variation and local-smoothness quantiles.Scoring statistics are detached and calibration is fixed; UID variants require at least 32 valid scoring tokens.Source: §B.1 · p. 14

B.2 Majority-Gated Online Optimization

  • Online majority-gated optimization uses majority-vote answers as pseudo-labels, then modifies rewards only among responses agreeing with the majority.Auxiliary scores do not affect answer selection, and the combined rewards are normalized within each prompt group.Source: §B.2 · p. 14

B.2.1 Effect of Auxiliary Rewards under Advantage Normalization

  • In unanimous groups, nonconstant auxiliary rewards can create normalized advantages where answer-only TTRL gives zero reward-based advantages, even when the absolute auxiliary reward scale is small.If auxiliary terms are identical, advantages remain zero; unanimous agreement may still be incorrect.Source: §B.2.1 · p. 15

B.3 Offline Training

  • Offline training compares score-selected correct-response preferences and weighted SFT, with methods matched to a shared response pool or prompt set within each objective.Preference pairs use a minimum score gap, while weighted SFT uses the same eligible responses across weighting methods.Source: §B.3 · p. 16

C.1 Details of the Training Dataset

The appendix specifies the online adaptation data, offline training pools, evaluation suites, and training and inference settings. It also reports warm-start SFT and UID-Mix comparisons, alongside a trajectory-permutation control.

C.1 Details of the Training Dataset

  • Online adaptation uses Qwen3-1.7B on all 30 AIME2026 problems, so AIME evaluation measures adaptation on the target benchmark rather than generalization to an unseen test set.Training rewards come from agreement with model-generated majority-vote pseudo-labels, not reference answers.Source: §C.1 · p. 16

C.2 Details of the Benchmark Dataset

  • The benchmark suite includes math and science evaluations plus transfer tasks; the 1,800-case transfer suite spans five benchmarks and uses different response-sampling protocols across tasks.The LabBench2 subset is conditioned on text-evidence availability and is not representative of its full multimodal benchmark.Source: §C.2 · p. 17

C.3 Hyperparameters and GPUs for Training

  • Online training uses LoRA-adapted GRPO, while offline weighted SFT and DPO use separate training configurations and objectives.The online setup runs for two dataset epochs; weighted SFT and the shared DPO warm-start SFT each run for one epoch.Source: §C.3 · p. 19

C.4 Hyperparameters for Evaluation

  • Mathematical benchmark accuracy averages five sampled responses rather than measuring pass@5 or majority-vote accuracy; CriticBench metrics assess verdict correctness, not critique quality.Mathematical scoring uses normalized boxed-answer matching and exact string comparison.Source: §C.4 · p. 20

D.1 Performance of Warm-start SFT for DPO training

  • Warm-start SFT raises AIME2026 and MinervaMath accuracy by 0.67 and 1.25 percentage points, respectively, but lowers HMMT2026 by 1.22 points, for a 0.23-point macro-average gain.The passage characterizes this as largely preserving baseline accuracy; the table caption identifies Table 10 as the base-versus-SFT comparison.Source: §D.1 · p. 20, Table 10 · p. 20

D.2 UID-Conj. Metric Analysis

  • UID-Mix raises the online four-benchmark macro-average from 50.46 to 54.50 for Qwen3-4B, while offline Qwen3-1.7B UID-Mix reaches 38.07 versus 38.43 for UID-Conj.Online GPQA-D decreases slightly, and the offline variants perform similarly; the comparisons do not establish a consistently superior aggregation.Source: §D.2 · pp. 20–21, Table 11 · p. 21

D.3 Isolating Trajectory-Specific Learning Signals

  • UID-Conj. outperforms the trajectory-permutation control on AIME2026, CriticBench F1, and CriticBench accuracy, but performs slightly worse on GPQA-D and MinervaMath.The comparison supports a trajectory-specific benefit that is not uniform across benchmarks.Source: §D.3 · p. 21

D.4 Paired Bootstrap Procedure in Uncertainty Analysis

The analysis estimates paired differences by resampling prompts within benchmarks and equally weighting benchmarks for its primary multi-benchmark results. It also reports prompt-weighted estimates as a distinct estimand and limits uncertainty claims to the evaluated checkpoints, generations, and procedures.

D.4 Paired Bootstrap Procedure in Uncertainty Analysis

  • Prompt-level paired bootstrap resampling preserves the five responses per method and uses the same sampled prompt indices for both methods to construct 95% intervals.Prompts are resampled within each benchmark, with benchmark identities and sample sizes fixed.Source: §D.4 · p. 21
  • The primary multi-benchmark estimate equally weights benchmarks, while the online and offline averages cover 503 and 533 prompts, respectively.The online average includes HMMT2026, MinervaMath, and GPQA-D; offline additionally includes AIME2026.Source: §D.4 · p. 21
  • Intervals are conditional on evaluated checkpoints, saved generations, benchmark composition, and scoring procedures; they omit training-seed and alternative-benchmark variation and do not adjust for multiple comparisons.An interval containing zero does not establish equivalence.Source: §D.4 · p. 22
  • Prompt weighting changes the weighted-SFT comparison from a macro-average difference of +1.93 percentage points [0.44, 3.44] to a prompt-weighted difference of +0.98 [−0.30, 2.29]; all four secondary intervals include zero.Benchmark-macro analysis remains primary; prompt weighting is reported as a distinct estimand, not a replacement chosen for narrower intervals.Source: §D.4 · p. 22

E.1 Preliminary Diagnostic vs. Training Behavior

The preliminary diagnostic distinguishes properties favored by SC and UID-Conj.; their differing training effects remain hypotheses whose downstream value varies by setting.

E.1 Preliminary Diagnostic vs. Training Behavior

  • SC scores higher on overall judged trace quality, validity, and consistency, while UID-Conj. scores better on relevance and non-redundancy.These are preliminary diagnostic results, not a claim that either preference is uniformly better for training.Source: §E.1 · p. 22
  • UID-Conj. favors traces with both global entropy variation and local smoothness, whereas SC summarizes predictive certainty without its temporal arrangement.The proposed explanation is that UID-Conj. can emphasize successful traces moving between more- and less-certain stages; the authors present this as a possible explanation, not an established mechanism.Source: §E.1 · p. 23
  • UID-Conj. can favor different successful trajectories without selecting the highest-judged response, while the downstream value of these preferences depends on the setting.Weighted SFT changes correct responses’ relative training contributions by auxiliary score, and DPO reinforces the highest-scoring correct response over the lowest-scoring alternative; links to aggregate outcomes remain hypotheses.Source: §E.1 · p. 23

E.2 Further Directions

  • Further work should test whether entropy-trajectory structure identifies useful updates beyond response length, confidence, and judged quality, and how effects vary with policy and objective.The passage also identifies interaction with online advantage normalization as an open question.Source: §E.2 · p. 23

F.1 Prompts used for Preliminary Diagnostic

  • The preliminary diagnostic generates step-by-step boxed-answer responses without reference answers, then has a judge assess each response independently on four criteria.The judge sees only the problem and one response, and may return a null score when a criterion cannot be assessed reliably.Source: §F.1 · p. 23

F.2 Prompts used for Training

  • Online rollouts exclude reference answers and majority-vote pseudo-labels from generation; offline DPO and weighted SFT share a step-by-step response pool.DPO uses chosen and rejected responses, while weighted SFT uses prompt–response examples with assigned weights.Source: §F.2 · pp. 23–24

F.3 Prompts used for Evaluation

  • Evaluation prompts vary by benchmark: mathematical tasks request boxed answers, while non-verifiable tasks preserve benchmark-specific inputs and instructions.CriticBench requests JSON verdicts; IHEval uses supplied tool outputs without live tool execution, and LABBench2 includes frozen supporting evidence.Source: §F.3 · p. 24

G Usage of AI Assistants

  • AI writing assistants were used for language refinement only, not technical content, methodology, or experimental analysis.The authors state that the scientific ideas, results, and conclusions were conceived and authored by the researchers.Source: G Usage of AI Assistants · p. 24
Loading 2610.41296v1…