PaperlayerView exact revision on arXiv

Source-linked AI summary

Weak-Driven Learning: How Weak Agents make Strong Agents Stronger

Zehao Chen, Gongxun Li, Tianxiang Ai, Zixuan Huang, Xiaodong Liu, Yifei Li, Wang Zhou, Fuzhen Zhuang, Xianglong Liu, Jianxin Li, Deqing Wang, Yikun Ban

arXiv:2602.08222v2cs.AI

TL;DR

Post-training can saturate when target predictions become highly confident, while useful supervision remains in historical weak checkpoints and their discrepancies with stronger states. WMSS identifies informative discrepancies through entropy-based selection and uses mixed-logit corrective training, producing gains across math, code, and logic without deployment-time inference changes. The method still requires extra training-time weak-agent computation and is most applicable when useful weak–strong discrepancies remain in structured reasoning and coding tasks.

  • Problem

    Existing post-training and self-improvement methods may require stronger external teachers, newly generated supervision, or additional computation, while target-only training can yield diminishing returns on hard samples.

  • Method

    WMSS reuses weak historical checkpoints, selects informative weak–strong entropy discrepancies with WD-DS, and applies mixed-logit corrective training with WD-JT.

  • Results

    WMSS achieves gains across math, code, and logic, with up to +5.9 points in Math-Avg and +4.4 points in Code-Avg over SFT, while requiring no deployment-time inference changes.

  • Takeaways & Limitations

    Historical weak states can be reused as corrective supervision for continued post-training, extracting more value from existing training trajectories without external strong teachers or deployment-time inference mechanisms.

  • Takeaways & Limitations

    WMSS adds a weak-agent forward pass during training, depends on useful weak–strong token-level discrepancies, and has been evaluated mainly on structured reasoning and coding tasks.

Abstract

from arXiv · show

As post-training optimization becomes central to improving large language models, we observe a persistent saturation bottleneck: once models grow highly confident, further training yields diminishing returns. While existing methods continue to reinforce target predictions, we find that informative supervision signals remain latent in models' own historical weak states. Motivated by this observation, we propose WMSS (Weak Agents Can Make Strong Agents Stronger), a post-training paradigm that leverages weak checkpoints to guide continued optimization. By identifying recoverable learning gaps via entropy dynamics and reinforcing them through compensatory learning, WMSS enables strong agents to improve beyond conventional post-training saturation. Experiments on mathematical reasoning and code generation datasets show that agents trained with our approach achieve effective performance improvements, while incurring zero additional inference cost.

1 Introduction

The paper introduces Weak-Driven Learning, which uses historical weak checkpoints to expose unresolved token-level errors and strengthen a current model without deployment-time inference overhead. WMSS combines discrepancy-aware selection with mixed-logit training and reports gains across mathematical reasoning and code generation.

  • Weak–strong discrepancies at key decision tokens reveal uncertain or incorrect alternatives that identify where the strong agent still needs sharper decisions.Routine tokens may provide little additional signal, whereas weak-revealed hard negatives mark unresolved boundaries.
  • WMSS uses mixed weak and strong logits with ground-truth cross-entropy so weak-model errors become token-level corrective signal for the strong branch.The strong branch is retained as the output model.
  • Weak-Driven Learning uses weak historical checkpoints as corrective training signals for strengthening a model on hard samples.The approach differs from directly imitating either a stronger teacher or the weak agent.
  • WMSS combines Weak-Driven Discrepancy Selection for entropy-discrepancy sample selection with Weak-Driven Joint-Logit Training for weak-driven logit mixing.Together, the modules operationalize weak-driven learning without additional inference overhead.
  • +5.9 points in Math-Avg and +4.4 points in Code-Avg are the reported maximum WMSS improvements over SFT across the reported backbones.The method nearly doubles the average gain over SFT compared with the strongest baseline while adding no deployment-time inference cost.

2 Related Work

Prior post-training methods commonly transfer knowledge through distillation, self-predictions, rollouts, or inference-time steering. WMSS instead reuses historical model states as corrective signals during post-training and avoids deployment-time search.

  • SFT and knowledge distillation are common post-training approaches, but distillation depends on stronger teacher models that may be proprietary, expensive, or domain-specific.These methods can leave hard or high-discrepancy samples insufficiently corrected.
  • Self-improvement methods often construct supervision through sampled responses, auxiliary contexts, alternate variants, preference optimization, or related model-derived signals.The related-work landscape includes weak-to-strong supervision and several self-improvement approaches.
  • WMSS differs by reusing historical model states as corrective signals during post-training rather than relying on inference-time steering, rollout curricula, or selection-only entropy scores.The distinction is the use of weak historical states for training-time correction without deployment-time search.

3 Weak-Driven Learning

Weak-Driven Learning improves a strong agent through systematic discrepancies with a weaker agent, while preserving the ground-truth task target as the optimization anchor. The weak agent contributes corrective information rather than serving as an imitation target.

  • Weak-Driven Learning drives strong-agent improvement through systematic discrepancies with a weaker agent instead of imitating a stronger teacher or the weak agent directly.The weaker agent may be a historical checkpoint or a related lower-capability model.
  • The agents can provide logits, probabilities, hidden representations, generated responses, rewards, uncertainty estimates, or other model-derived signals.The framework is defined broadly over possible weak–strong outputs.
  • Weak-driven learning constructs discrepancy-aware supervision that transforms weak–strong differences into corrective training signals while preserving the task target y.The abstract signal is SWD = Φ(ostrong(x), oweak(x), y).
  • The post-training objective optimizes both agents while retaining only the updated strong agent for downstream use.The weak agent participates in training but is not the final deployed model.
  • Weak-agent errors, uncertainty, and disagreement expose residual weaknesses or under-optimized regions of the strong agent, converting discarded training artifacts into corrective supervision.The weak agent is therefore useful because its mistakes reveal areas requiring further optimization.

4 Method: WMSS

WMSS operationalizes weak-driven learning with two modules: WD-DS selects informative examples using weak–strong entropy discrepancies, and WD-JT trains paired agents through mixed logits. The framework initializes a weak historical checkpoint, forms an active dataset, and retains the strong branch after training.

  • WMSS overview: WMSS combines WD-JT, which converts weak-model confusion into gradient signal through logit mixing, with WD-DS, which selects hard or high-discrepancy samples.WD-JT is the core mechanism, while WD-DS determines where its computation is most useful.
  • Initialization: Phase 1 performs SFT from base model M0 to obtain M1 and initializes the weak agent from the historical checkpoint M0.The weak agent then provides logits during subsequent training.
  • WD-JT: WD-JT mixes strong and weak logits as zmix(x) = λ zstrong(x) + (1 −λ) zweak(x), then optimizes the resulting distribution with ground-truth cross-entropy.The mixed distribution exposes weak-revealed hard negatives rather than treating the weak agent as a teacher to imitate.
  • Iterative training: The iterative loop computes entropy differences, samples and deduplicates an active dataset, applies mixed-logit training, and returns the retained strong branch.The three phases are initialization, WD-DS, and WD-JT.
  • WD-DS: WD-DS prioritizes samples that remain learnable and show weak–strong discrepancies while downplaying samples confidently handled by both agents or lacking useful disagreement.Its three terms select consolidation samples, currently difficult samples, and recoverable regressions.

5 Mechanistic Analysis of Weak-Driven Learning

WMSS explains weak-driven learning as a mechanism that increases corrective gradients on weak-revealed hard negatives by mixing weak and strong logits. Under explicit margin conditions, this shifts probability mass toward negative tokens while preserving a ground-truth cross-entropy objective.

  • Gradient mechanism: WMSS increases probability mass on weak-revealed hard negatives, strengthening their token-level corrective gradients.Target-only SFT can assign very small probabilities to unresolved hard negatives, leaving limited gradient signal.
  • Mixed-logit training: The mixed-logit objective fuses weak and strong logits on the same batch and computes cross-entropy against the ground-truth token.The strong branch remains the output model while weak logits expose plausible hard negatives.
  • Hard-negative selection: Hard negatives are tokens whose weak-agent target margins are smaller than the strong agent’s margins, indicating less separation and greater confusion.The mixed margin is a convex combination that shrinks margins toward the weak agent on this set.
  • Conditional analysis: When the weak agent is uniformly more uncertain than the strong agent, mixed-logit training shifts probability mass from the target to negative tokens.In this sufficient regime, negative-class gradient magnitudes can increase on hard samples.
  • Conditional analysis: The local update analysis shows negative-token changes scale with mixed negative probability, while target-logit changes scale with one minus mixed target probability.The full parameter-space update additionally passes through the Jacobian Gram matrix.

6 Experiments

Experiments compare WMSS with post-training baselines under a shared two-epoch budget across mathematical reasoning, code generation, and logical reasoning. WMSS improves benchmark averages across tested backbones, while module ablations and trajectories show complementary module roles and diminishing late-stage utility.

  • Main comparisons: +4.6 points: WMSS’s average Math Avg improvement over two-epoch SFT exceeds the +1.2-point average improvement of non-WMSS baselines.On the three Qwen math blocks, this is more than 3× the mean competing gain.
  • Main comparisons: +5.9 Math Avg points: Qwen3-8B-Base improves from 66.9% to 72.8% over the two-epoch SFT baseline.Other tested bases also improve: Qwen3-4B-Base by +4.9 and Qwen2.5-3B by +2.9.
  • Main comparisons: WMSS improves Math Avg on every tested base across Qwen3-4B, Qwen3-8B, Qwen2.5-3B, and Gemma-3-4B-PT.The reported gains are +4.9, +5.9, +2.9, and +4.4, respectively.
  • Module ablations: 59.9%: combining WD-DS and WD-JT outperforms WD-DS alone at 56.3% and WD-JT alone at 58.2% on the Qwen3-4B-Base ablation.The full combination also improves AIME2025 from 13.3% to 20.0%.
  • Convergence dynamics: Extended four-epoch trajectories show early gains followed by stabilization, with AMC23 regression and GSM8K volatility during late-stage over-optimisation.These trajectories are diagnostics beyond the fixed two-epoch main comparison, not a causal early-stopping rule.
  • Step reduction: 123 steps: WD-DS reaches 0.714/0.876 on MATH500/GSM8K, versus 0.694/0.873 without WD-DS using 218 steps.The active schedule uses about 56% of the full budget and surpasses the comparison schedule at early checkpoints.

7 Discussion

The discussion frames historical weak checkpoints as reusable corrective supervision for post-training. WMSS combines weak-driven logit mixing with discrepancy-aware selection and reports gains without online rollouts, external strong teachers, or deployment-time inference changes.

  • Discussion: Historical weak states can provide corrective training signal instead of being treated as obsolete checkpoints.Weak agents expose plausible but wrong alternatives on hard tokens while ground-truth cross-entropy anchors optimization to the correct answer.
  • Discussion: WMSS reuses training artifacts and requires no online rollouts, external strong teachers, or deployment-time inference changes.The framework combines weak-driven logit mixing with discrepancy-aware selection to focus weak-agent computation where differences are informative.

A Limitations and Broader Implications

WMSS has no deployment-time inference overhead but requires an additional weak-agent forward pass during training and depends on useful weak–strong discrepancies. The reported evidence focuses on structured reasoning and coding, with broader open-ended generation left as future work.

  • WMSS adds no inference-time overhead but requires an additional weak-agent forward pass during training.
  • The method is most attractive when modest extra training computation is acceptable in exchange for stronger deployment performance.
  • WMSS relies on useful token-level discrepancies between weak and strong agents; corrective signal diminishes when the agents become nearly identical.
  • Experiments focus on structured reasoning and coding tasks, where sharpening decisions against hard negatives aligns with evaluation objectives.
  • Extending WMSS to open-ended generation may require task-adaptive mixing or softer objectives.

B.2 Results on Gemma-3-4B-PT

On Gemma-3-4B-PT, WMSS achieves the best reported Math-Avg and Code-Avg relative to SFT, while diagnostics support suppression of non-target logits as its mechanism. Sensitivity analyses indicate broad competitive hyperparameter regions, but harder-task gains depend on regression repair.

  • Results on Gemma-3-4B-PT: +4.4 Math-Avg and +3.9 Code-Avg over SFT are achieved by WMSS on Gemma-3-4B-PT.
  • Results on Gemma-3-4B-PT: WMSS attains the best LogiQA 2.0, ReClor, and logic-average accuracy across all four base models.
  • Mechanistic diagnostics: The non-target logit mean falls from 2.09 to 0.90 (−56.9%), while the target logit changes from 35.88 to 36.10.
  • Mechanistic diagnostics: Small entropy gaps provide little corrective signal, whereas sufficient separation helps without continuously improving as the gap grows.
  • Sensitivity analysis: The λ sweep has a broad plateau over λ ∈[0.42, 0.50], and the fixed default lies within about 2 percentage points of neighboring configurations.
  • Sensitivity analysis: Disabling regression repair raises MATH 500 to 70.2% but lowers AIME 2025 to 10.3%, while the default reaches 16.7% on AIME.

B.8 Sensitivity Analysis on Mixing Coefficient λ

The mixing coefficient λ balances weak-agent corrective signal against strong-agent target fitting. Performance follows a broad inverted-U region rather than a sharp optimum, with degradation when either branch dominates.

  • λ explicitly controls the relative weight of weak and strong logits in mixed-logit training.
  • The best average performance is 75.5% at λ = 0.42, with a strong plateau across λ ∈[0.42, 0.50] including the fixed default λ = 0.5.
  • At λ = 0.9, average performance falls to 67.6% as the mixed logits rely mainly on the strong model.
  • At λ = 0.3, a weaker model reaches only 13% accuracy on MATH500 because weak-model dominance reduces effective target learning.
  • The optimum near λ ≈0.42 preserves weak-agent probability mass on hard negatives while retaining strong-agent target fitting.
  • The theoretical crossover is a local consistency check, not an accuracy inversion rule, and predicts a broad optimum because sensitivity changes during training.

B.9 Pass@k Comparison with DPO

WMSS outperforms DPO most clearly on hard reasoning benchmarks at low sampling budgets, while DPO provides larger diversity gains as k increases. The comparison and accompanying analysis connect this pattern to weak-agent corrective gradients on hard negatives.

  • Performance comparison: WMSS beats DPO at every k on AIME 2025 and AMC23, including pass@1 gains of +3.8 and +6.7 percentage points.
  • Performance comparison: On MATH500, WMSS leads at pass@1 and pass@4, while DPO catches up for k ≥8 within 0.4 percentage points.
  • Interpretation: The pattern is consistent with WMSS helping where hard-reasoning gradients remain informative, while preference-based diversity narrows gaps on easier benchmarks.
  • Performance comparison: The relative WMSS–DPO gap shrinks as k grows, from 1.27× to 1.11× on AIME 2025 and from 1.16× to 1.03× on AMC23.
  • Diversity analysis: DPO has higher diversity gains across all benchmarks, but its additional sampling does not close the absolute gap on AIME 2025 or AMC23 by pass@64.

D Additional Dynamics Diagnostics for Mixed-Logit Training

The appendix analyzes how mixed-logit training changes branch gradients, hard-negative probability, and convergence behavior, while identifying validity limits and late-training drift.

  • Setup: WD-JT mixes weak and strong logits, jointly backpropagating through both branches while retaining the strong branch for evaluation.The mixed objective uses the paired logits and ground-truth cross-entropy.
  • Hard-negative amplification: Under local linearization, weak-model confusion can shrink hard-negative margins and increase negative probability mass, amplifying corrective gradients.This mechanism is presented as a sufficient local regime rather than a general guarantee.
  • Strong-model dominance: The strong branch dominates the effective mixed-logit update under an explicit sensitivity-advantage assumption tied to greater sharpness and lower entropy.The analysis characterizes branch sensitivity rather than predicting an accuracy inversion.
  • Gradient shielding: When the fused prediction becomes nearly one-hot, residual gradients and cross-branch sensitivity collapse, producing gradient shielding that suppresses weak-induced corrections.This explains why the weak agent’s interaction becomes increasingly shielded as confidence rises.
  • Null-space drift: Global logit means can drift without changing predictive distributions or centered sharpness because cross-entropy has no restoring force along the all-ones direction.Optimizer noise and regularization can move parameters in this shift-invariant null space.
  • Validity and failure modes: The analysis is limited by sufficient margin assumptions, single-step linearization, and evaluation on single-mode tasks where aggressive suppression may be inappropriate for multi-completion settings.Violating tokens can reduce or reverse intended amplification locally, while late training can accumulate raw mean-logit drift.

E.1 Evaluation Benchmarks and Baselines

The evaluation spans mathematical reasoning, code generation, and logical reasoning, using verified training data, multiple baselines, and matched main-comparison budgets. WMSS follows a one-epoch SFT warm-up with one weak-driven epoch, while extended training is treated as diagnostic because over-optimization can occur.

  • Benchmarks: The evaluation covers seven mathematical benchmarks, four code-generation benchmarks, and two logical-reasoning benchmarks.Math includes AIME2025, MATH500, AMC23, AQuA, GSM8K, MAWPS, and SVAMP; code includes HumanEval+, MBPP+, BigCodeBench, and LiveCodeBench.
  • Baselines: Baselines include standard SFT, UNDIAL, NEFTune, SPIN, and SSB, representing conventional training, perturbation, stochastic penalties, self-play, and self-distillation approaches.The baselines differ in whether they use logit penalties, embedding noise, generated preference pairs, or hint-conditioned soft targets.
  • Training data: The three-domain training mixture uses verified samples in chat format, with mathematics forming the largest pool at approximately 5.7 times code and 7.5 times logic.Assistant completions retain full chain-of-thought and structured final answers.
  • Training data: The math, code, and logic training sets contain 111,657, 19,457, and 14,882 verified examples, respectively, after domain-specific filtering and answer verification.The logic set retains 98.3% of candidates after held-out evaluation splits are removed.
  • Training configuration: Main SFT-style comparisons use a fixed two-epoch budget, with WMSS allocating one SFT warm-up epoch followed by one weak-driven epoch.WMSS then applies WD-DS and WD-JT during the second epoch.
  • Training configuration: Extended convergence diagnostics can reveal over-optimization on small benchmarks without improving the math average, so they are not used as benchmark-specific early-stopping evidence.The reported main comparisons retain the fixed budget rather than selecting training duration per benchmark.
Loading 2602.08222v2…