Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization

1 Introduction

• Conventional pipelines retain positive trajectories, but discarded negative traces may contain valid intermediate reasoning and diverse modes useful for OOD transfer.The introduction frames poor generalization and data inefficiency as limitations of existing SFT practices.Source: §1 · p. 1
• Negative chains contain 22 recurring patterns, while negative-only SFT shows slower convergence and higher policy entropy and pass@k on OOD tasks.The paper links these patterns to reduced shortcut overfitting during training and more diverse inference exploration.Source: §1 · p. 2
• GLOW uses the entire dataset and adaptively upweights samples with low inter-epoch loss reduction to prioritize undercovered reasoning patterns.The paper describes this as a sample-aware reweighting strategy based on learning progress.Source: §1 · p. 2
• 5.51% OOD performance gain over positive-only SFT and 2.14% average-performance gain over mixed-data training; GLOW also raises MMLU from 72.82% to 76.47% as an RL initialization.The MMLU comparison uses the same RL setup.Source: §1 · p. 2
• The paper reports that GLOW improves OOD generalization across diverse benchmarks and transfers effectively to RL initialization.The introduction characterizes GLOW as amplifying gains from RL.Source: §1 · p. 2

2 Related Works

• SFT is widely used to strengthen LLM reasoning, with teacher-distilled chain-of-thought trajectories commonly used to supervise smaller students (Wei et al., 2021; Ouyang et al., 2022; Shao et al., 2024a; Zheng et al., 2025; Yu et al., 2025b).Rejection sampling commonly keeps only trajectories with correct final answers, discarding other supervisory information.Source: §2 · pp. 2–3
• Prior negative-data methods use prompting, rewrite incorrect traces into positive supervision, or mark correct and incorrect samples, but do not test raw incorrect trajectories as generalization supervision.Prompt-based approaches also depend on existing instruction-following and reasoning ability.Source: §2 · p. 3
• Domain-generalization studies diagnose limited cross-domain transfer but do not develop methods or examine how data selection and supervision signals affect it.The cited work discusses domain-specific SFT and superficial reasoning traces as possible concerns.Source: §2 · p. 3

3.1 Data Construction and Training Setup

• The controlled setup distills Qwen3-8B trajectories from OpenMathReasoning and MMLU, labels ground-truth matches positive and other samples negative, and fine-tunes Qwen-2.5 models from 3B to 32B and Llama-3.1 8B.The datasets contain balanced sets of complete reasoning chains.Source: §3.1 · p. 3

3.2 Negatives Surpass Positives in OOD

• 11.97% average improvement on general reasoning tasks and 4.11% on other reasoning tasks followed negative math-sample training; negative MMLU-sample training yielded 1.98% on mathematical reasoning and 1.35% on other reasoning benchmarks.These are reported OOD gains; in-domain improvements from negative samples were smaller than those from positives.Source: §3.2 · p. 3
• Negative-sample training consistently improves OOD generalization across benchmarks, including when training data comes from either math or MMLU.The paper reports the same trend for MMLU-trained models, so the observed benefit is not limited to a specific domain.Source: §3.2 · p. 3

4.1 Data Perspective

• Negative samples span nine major error types and 22 sub-types, covering more heterogeneous reasoning patterns than the more uniform positive solution templates.The categories include logical mistakes and comprehension errors.Source: §4.1 · p. 5, Table 3 · p. 5
• Negative trajectories improve OOD generalization by exposing models to diverse error regimes that favor invariant reasoning features.Negatives can retain partially valid reasoning, while positives cover fewer paths and offer weaker incentives for stability across regimes.Source: §4.1 · p. 5

4.2 Training Perspective

• Negative-only SFT descends more gradually than positive-only SFT, while its loss and benchmark performance continue improving across training.The smaller early loss reductions and sustained learning signal are consistent with reduced shortcut fitting rather than noise.Source: §4.2 · p. 5

4.3 Inference Perspective

• Mneg maintains higher entropy throughout the thinking span and a sharper decline at the answer boundary; on OOD data, Mpos’s entropy margin reverses.Off-policy entropy is inherently higher because teacher forcing can expose contexts that have low probability under a model’s own policy.Source: §4.3 · p. 6
• Compared with Mpos, Mneg uses more discourse and hesitation tokens, fewer numerals, and maintains more plausible continuations before settling on an answer.The authors describe this as allocating more generation budget to connective exploration before concrete computation.Source: §4.3 · p. 6

5.1 GLOW: Gain-Based Loss Weighting

• GLOW measures sample gain by inter-epoch loss reduction and upweights low-gain examples, steering training toward weakly covered patterns.Small gain indicates limited effective coverage under the current trajectory.Source: §5.1 · pp. 6–7

5.2 Discussion with Prior Works

• Unlike memoryless easy-sample downweighting, GLOW targets stagnant-progress examples to cover underexplored reasoning patterns.Prior SFT reweighting uses current loss to down-weight easy samples; GLOW instead targets coverage.Source: §5.2 · p. 8

5.3 Experimental Results

• GLOW reaches 55.18 in-domain and 67.25 OOD on Qwen2.5-7B, with consistent in-domain gains across scales and superior OOD results.The section reports cross-domain performance for math-trained models and general-reasoning models.Source: Table 6 · p. 7, Table 7 · p. 7, §5.3 · p. 8
• GLOW prioritizes negatives during early training, suggesting its gain-based weights target harder and underrepresented reasoning patterns.The prioritization is reported for early training stages.Source: §5.3 · p. 8
• GLOW increases thinking-span entropy while answer-span entropy remains stable or decreases under OOD, pairing broader reasoning exploration with relatively decisive answers.Reported examples include thinking entropy changes from 0.36 to 0.71 for Math to Math and 0.96 to 1.44 for MMLU to Other domains.Source: §5.3 · p. 8, Table 8 · p. 8
• With RL data, optimizer, and hyperparameters fixed, GLOW remains superior after RL, supporting its stronger SFT initialization.The comparison varies the SFT objective to isolate its impact on subsequent RL post-training.Source: §5.3 · p. 8, Table 9 · p. 8

• Negative reasoning trajectories can improve SFT generalization and mitigate OOD degradation.Source: §6 · p. 8
• Data, training, and inference analyses examine why negatives help and how they shape optimization and model behavior.Source: §6 · p. 8
• GLOW upweights undercovered examples using inter-epoch loss reduction, yielding data-efficient training and consistent cross-domain gains.Source: §6 · p. 8

• The study examines gain-based reweighting during supervised fine-tuning; its interaction with later RLHF or other RL stages remains future work.Source: Limitations · p. 9
• Experiments focus on text-only CoT data for math and multi-task knowledge benchmarks.Source: Limitations · p. 9
• Experiments use a small set of open-source backbones, with broader task families, scales, multimodal, and tool-augmented settings left for future study.Source: Limitations · p. 9

A.1 Experiments Setup

• The data contains 7.2k math and 17.4k commonsense instances, sampled in a 1:1 positive-negative ratio after removing trajectories longer than 8,192 tokens.Trajectories are labeled by whether the final answer matches ground truth.Source: §A.1 · p. 12
• Experiments fine-tune Qwen2.5 models from 3B to 32B and LLaMA-3.1-8B for 20 epochs, using batch size 128 and a maximum learning rate of 5×10−5.The stated reason for 20 epochs is that loss does not converge earlier and benchmark performance continues improving through that point.Source: §A.1 · p. 12
• Evaluation covers math, commonsense, and other OOD reasoning benchmarks; OOD domains are defined relative to the training-data distribution.Accuracy is the reported performance measure.Source: §A.1 · p. 12

A.2 Detailed Theoretical Derivation

• The framework motivates GLOW’s dynamic reweighting mechanism through an idealized analysis of optimization conditioning.The supplied passage does not specify the assumptions or derivation beyond noting they are idealized.Source: §A.2 · p. 12
• Inter-epoch loss reduction serves as a proxy for how strongly a sample’s gradient aligns with the current update direction.Small-gain samples are treated as weakly aligned with recent optimization and thus as candidates for upweighting.Source: §A.2 · p. 12
• Under the framework’s assumptions, upweighting low-gain samples can add curvature in complementary directions and improve local conditioning.The argument depends on low-gain samples aligning with undercovered subspaces.Source: §A.2 · p. 12

A.2.7 Conditioning and Stability-Based Generalization

• Under stated assumptions, gain-based reweighting adds positive semidefinite Fisher curvature and can improve conditioning on low-curvature directions.The argument links smaller loss gains to weaker alignment with recent updates; positive semidefinite augmentation alone does not ensure better curvature in bottleneck directions.Source: §A.2.3 · p. 13, §A.2.4 · p. 13, §A.2.5 · p. 13

A.3 IRM View of Diverse Negative Trajectories

• Negative error categories are treated as distinct IRM environments; the training errors span nine major categories and 24 subcategories.The authors interpret this diversity as providing signals that remain predictive across heterogeneous error environments.Source: §A.3 · pp. 14–15, §A.4 · p. 15

A.5 Hyperparameter Sensitivity of GLOW

• Across tested settings, GLOW modestly improves on the full-SFT reference; β = 12 is generally stronger than β = 10 or β = 18 at matched α.Changing α from 0.8 to 1.5 leads to small performance changes.Source: §A.5 · p. 15, Figure 4 · p. 15

A.7 Model Performance Evolution Across Epochs

• Across Qwen2.5-7B and 32B checkpoints, negative-trajectory SFT beats the base model and often matches or exceeds its positive-trajectory counterpart on OOD benchmarks.The cited table covers epochs 5, 10, 15, and 20; the supplied passages do not provide specific checkpoint scores.Source: §A.7 · pp. 15–16, Table 10 · p. 16

A.8 Negatives Are Frequently Upweighted by GLOW

• Negatives account for more than 50% of the most-upweighted examples in most epochs, peaking around 75%–80% early before approaching 50%.The analysis selects the example with the largest upweighting signal at each optimization step.Source: §A.8 · p. 16, Figure 6 · p. 18

A.9 Pass@k under OOD Evaluation

• Negative-trained models consistently achieve higher OOD pass@k than positive-only models across all tested k.The comparison averages results over BBH, ACPBench, and HeadQA for OpenMathReasoning, and Olympia, ACPBench, and HeadQA for MMLU.Source: §A.9 · p. 16

A.6 Training Loss on OpenMathReasoning and MMLU

• Training losses fall across epochs, with positive-trajectory training converging faster than negative-trajectory training.The figure compares Qwen2.5 models and Llama3.1-8B on OpenMathReasoning and MMLU.Source: Figure 5 · p. 17

A.11 Case Study of Negative Samples

• Negative trajectories show more high-entropy reasoning fragments and more hesitation or discourse tokens, such as “wait” and “but,” than positive trajectories.The authors describe these tokens as accompanying more exhaustive exploration before finalizing a response.Source: §A.11 · p. 18, §A.12 · p. 18, Figure 10 · p. 20