Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

1 Introduction

• Reasoning-oriented post-training can improve target reasoning while regressing non-target abilities such as perception, safety, and factual grounding, especially during prolonged RL without explicit regularization.The introduction connects this risk to reported hallucinations and greater vulnerability to jailbreak attacks.Source: §1 · p. 2
• Reasoning-tuned open models often lag their base counterparts on perception and robustness, with a consistent VisOnlyQA drop across both Qwen2.5-VL families.MiMo-VL-7B-RL performs competitively with its base model on non-reasoning tasks; its detailed training strategy is undisclosed.Source: §1 · p. 2
• Reasoning-only training on Qwen2.5-VL-7B drops perception performance by 7%, while some reward signals converge up to three times faster than others.These different convergence rates complicate assigning weights across heterogeneous objectives.Source: §1 · pp. 2–3
• RECAP replays general-capability data during RLVR and dynamically reweights objectives according to convergence and instability, shifting emphasis away from saturated objectives.Its overview describes initially uniform objective weights followed by convergence-based adjustment.Source: §1 · p. 3, Figure 3 · p. 3
• The plug-in scheduler is magnitude-agnostic, uses no auxiliary models, and integrates into RLVR pipelines with negligible overhead.The introduction also characterizes it as end-to-end.Source: §1 · p. 3
• RECAP preserves or improves general capabilities while matching or exceeding reasoning-only finetuning in the reported experiments.The authors also report that it outperforms continual-finetuning baselines and is competitive with specialized open models while using less compute.Source: §1 · p. 3

2 Related Work

• RLVR uses verifiable signals in reasoning pipelines, while process-based reward models supervise intermediate progress but can invite reward hacking.The cited work describes agents exploiting the appearance of a correct process rather than achieving the intended outcome.Source: §2 · p. 4
• Continual-learning methods mitigate forgetting through parameter regularization, output distillation, or replay, while RLHF commonly uses a KL penalty to stabilize updates.InstructGPT also interleaves pretraining gradients with RLHF updates to reduce drift relative to KL-only regularization.Source: §2 · p. 4
• The paper frames catastrophic forgetting as regression on previously acquired skills and reviews approaches that constrain parameter changes, outputs, or training through replay.Its discussion contrasts these approaches with KL regularization and pretraining-gradient interleaving in post-training.Source: §2 · p. 4
• Some newer methods use verified rollouts, augmented-data discrepancy penalties, mixed verifiable reward suites, or reflection and re-attention to address forgetting.The authors state these methods do not guarantee performance preservation in non-target domains; reduced or removed KL in some reasoning pipelines may exacerbate forgetting.Source: §2 · p. 4

3 Background

• The paper adopts GRPO as its primary RL algorithm because it reduces memory and compute overhead relative to standard PPO.The background places GRPO within reasoning-centric language-model post-training.Source: §3 · p. 4
• SFT optimizes instruction–response likelihood and commonly initializes later preference- or reward-based alignment.The background describes SFT as supporting downstream instruction following and broad zero-shot generalization.Source: §3 · p. 4
• RLHF fits a reward model from pairwise preferences, then optimizes reward while regularizing toward a reference policy, commonly with a KL penalty.The described pipeline improves helpfulness and harmlessness while retaining base-model competence.Source: §3 · p. 4
• RLVR replaces learned preference rewards with verifiable signals such as exact-match answers, execution checks, or constraint checkers.The background says this reduces labeler noise and can better target reasoning fidelity while retaining a KL-regularized RL form.Source: §3 · p. 5
• GRPO computes advantages from group-normalized sequence rewards rather than a learned critic, and maximizes a clipped objective with KL regularization.The background describes it as useful when long chain-of-thought rewards are sparse and verifier-based.Source: §3 · p. 5
• The paper treats retention of pretrained general abilities and gains across reasoning datasets as joint concerns in modern post-training.General abilities include perception, grounding, instruction following, and safety.Source: §3 · p. 5

4 Our Method: Replay-Enhanced CApability Preservation (RECAP)

• RECAP combines replay of general-capability data with online objective reweighting based on local progress and instability signals.The method is described as operating without changing the underlying RL algorithm.Source: §4 · p. 5
• Per-objective convergence is estimated from current and preceding sliding-window loss averages because individual RL steps are unstable.The instability signal uses the coefficient of variation within a window.Source: §4 · pp. 5–6
• In the illustrated training setting, the format reward saturates quickly while reasoning accuracy fluctuates most, directing more priority toward reasoning.The figure caption describes format reward convergence and instability after 50 steps; its reported setting is the first experimental setting.Source: Figure 4 · p. 6
• A temperature-controlled softmax converts per-objective priority scores into normalized coefficients, with lower temperatures producing sharper priorities.The experiments use T = 5 by default.Source: §4 · p. 6
• The weighted objective favors objectives that converge slowly or fluctuate, while reducing weight on objectives that are already well learned and stable.As T approaches infinity, the weighting returns to equal weighting across objectives; the framework covers RL reward surrogates and supervised losses.Source: §4 · pp. 5–6
• RECAP’s softmax weighting is the unique solution to an entropy-regularized priority-allocation problem, rather than merely a heuristic normalization.The entropy term discourages degenerate allocation to one objective, while temperature controls the trade-off between sharper and smoother mixtures.Source: §4 · p. 7

5.1 Experimental Settings

• The experiments evaluate Qwen2.5-VL-3B in an RLVR-only setup and Qwen2.5-VL-7B in a hybrid RLVR/SFT setting with replayed perception datasets.The hybrid setup uses ThinkLite-VL-70k alongside RefCOCO and LLaVA-OneVision OCR; the study disables reference-KL by default to isolate replay effects.Source: §5.1 · pp. 7–8

5.2 Experimental Results

• RL training on the reasoning domain improves base-model performance on both reasoning and perception benchmarks, and Qwen2.5-VL-3B rises from 6 to 60 on SCIENCEQA.On SCIENCEQA, RECAP also outperforms the comparison open-source reasoning models.Source: §5.2 · p. 8, Table 1 · p. 8
• RECAP achieves the best or runner-up performance across hybrid benchmarks except VizWiz, where it improves the base model by more than 10%; it also obtains the highest segmentation score.Naive ThinkLite-VL-70k finetuning lowers LISA segmentation from 65.13 to 57.58, while replay baselines retain about 64.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 Ablation Studies

• With identical training pipelines, RECAP’s reweighting lets accuracy overtake the uniform baseline after step 40, while the baseline initially raises format reward faster.The scheduler down-regulates fast-converging, low-instability format rewards and reallocates weight toward harder, higher-variance objectives such as accuracy.Source: Figure 5 · p. 9, §5.3 · p. 10
• Applying a uniform thinking reward to segmentation rapidly reduces response length from several hundred characters to tens or nearly zero, motivating task-specific reward formats.The authors reserve thinking rewards for tasks that benefit from step-by-step reasoning and use answer-format rewards for perception domains.Source: §5.3 · p. 10, Figure 6 · p. 10
• On reasoning tasks, RECAP reduces average thinking length from 67 to 27 words while preserving accuracy, a 60% reduction that lowers inference latency and compute cost.The reported curves also show more stable variance and concise rationales.Source: §5.3 · p. 10, Figure 7 · p. 10

• RECAP replays general data during finetuning and reweights objectives without training external models, preserving general knowledge while improving target reasoning performance.Source: §6 · p. 10

• Empirical evaluation covers RLVR and standard SFT only; preference, alignment, and process-reward settings remain for future evaluation.Source: Limitations · p. 11

• The appendix provides training-dataset statistics, implementation details, and additional experimental results.Source: A Experimental setup · p. 18

A.1 Proof of Entropy-Regularized Priority Allocation

• The optimization problem admits at least one maximizer, and every maximizer lies in the simplex interior.The existence argument uses continuity on a compact set; the interior result rules out zero coordinates.Source: §A.1 · p. 18
• At an interior maximizer, the proof derives first-order optimality conditions under the simplex equality constraint.The proposition states that a scalar multiplier applies across all K coordinates.Source: §A.1 · p. 19
• For T > 0, the entropy-regularized priority-allocation problem has a unique maximizer.The proof establishes uniqueness using strict concavity on the simplex interior.Source: §A.1 · pp. 19–20
• The RECAP loss weights are defined from the closed-form optimizer and satisfy a mean-normalization property.The proof obtains the property by substituting the optimizer and using that its probabilities sum to one.Source: §A.1 · p. 20

A.2 Data statistics and implementation details

• OCR-focused subsets from the LLaVA-OneVision release and release-provided synthetic or curated data form the OCR corpus; images are resized to at most 512 pixels on the longer side.The resizing preserves aspect ratio and is intended to mitigate out-of-memory errors without changing task semantics (Li et al., 2024).Source: §A.2 · p. 20, Table 3 · p. 20
• Training uses GRPO and SFT losses with AdamW, a linear learning-rate schedule, and window size W = 10 and temperature T = 5.0.The learning rate warms up for 10% to 1×10^-6 and then decays linearly; the RLVR setting uses α = 0.5 without additional large-scale hybrid hyperparameter tuning.Source: §A.2 · p. 20
• The larger hybrid setup trains for 500 steps on 8 GPUs with effective batch size 16 and 64 rollouts per optimizer step.The setup uses data parallelism, per-device batch size 1, and two gradient-accumulation steps.Source: §A.2 · p. 21
• The larger-model evaluation spans multimodal understanding, visual reasoning, math-in-vision, OCR, and accessibility benchmarks, while the smaller setup uses a separate six-benchmark suite.The larger suite includes LISA, MMMU-Pro, AI2D, MathVista, MathVision, MathVerse, MMBench, VizWiz, and OCRBench v2; the smaller suite includes SAT, ScienceQA, MathVista, ChartQA, InfoVQA, and MMMU (p46, p47).Source: §A.2 · p. 21

Evaluation prompt

• Prompts request reasoning and final answers in tagged fields, while qualitative examples show increasingly concise, direct responses on simple visual tasks.During training, chains of thought on image-understanding tasks shorten toward direct answers, motivating non-thinking rewards when reasoning is unnecessary.Source: Evaluation prompt · p. 21, Figure 8 · p. 24, §A.4 · p. 25

A.6 Reward dynamics during training

• Rewards fluctuate substantially, with total-reward standard deviation peaking near 0.9 around step ∼20, motivating sliding-window signals; W = 10 balances responsiveness and robustness.W = 25 delays reweighting until 50 iterations into the 500-step Qwen2.5-VL-7B run, when some rewards are near-converged and convergence-rate signals are less useful.Source: §A.6 · p. 27, Figure 10 · p. 27

A.7 Reward values at the end of training

• For the five hybrid objectives, RECAP focuses least on format rewards, then IoU, next-token prediction on OCR, and most on reasoning accuracy.The coefficients are relatively consistent during the last 100 iterations; supervised finetuning and accuracy rewards receive emphasis due to instability.Source: §A.7 · p. 28, Figure 11 · p. 28
• +2.01 reasoning score, +1.11 IoU, and +1.40 mean token accuracy over the uniform baseline, with near-parity on thinking-format and direct-answer rewards.These results align with down-weighting saturated format signals and shifting weight toward harder, higher-variance objectives while maintaining output format.Source: §A.7 · p. 28, Figure 12 · p. 28
• RECAP closely tracks the Pareto front under noisy objectives and improves across all initializations, while CAGrad and Nash-MTL converge more slowly and cost more per step.The comparison uses a synthetic two-task problem based on the illustrative setup of Navon et al. (2022).Source: §A.7 · p. 29, Figure 13 · p. 30
• On NYUv2, RECAP nearly matches single-task baselines (∆m% ↓≈0), is roughly 3× more efficient in runtime and memory, and outperforms loss-reweighting methods except on Angle Distance Mean versus Uncertainty Weighting.It also surpasses several gradient-based methods while remaining roughly three times faster than gradient-based alternatives.Source: Table 4 · p. 31, §A.7 · p. 31
• With K = 3, gradient-based methods take about 300s versus about 100s for single-task baselines and loss-reweighting; in RLVR with K = 8, gradient manipulation would be roughly 8× slower.These methods incur Θ(K) time and space overhead by storing and computing all task gradients.Source: §A.7 · p. 32, Figure 15 · p. 32
• Using only convergence rate (α = 1) destabilizes the second objective, while using only inverse signal-to-noise ratio (α = 0) destabilizes the first; α = 0.5 or 0.75 yields more stable convergence.The intermediate trade-off values balance the two signals across initializations.Source: Figure 16 · p. 32, §A.7 · pp. 32–33
• Low temperature produces strong trajectory fluctuations near the Pareto front, indicating unstable training.Figure 17a shows the fluctuations near the front.Source: §A.7 · p. 33
• At T = 30, the two initializations farthest from the Pareto front require many more steps to approach it.Their trajectories remain red for longer.Source: §A.7 · p. 33
• For T > 1, temperature regularizes against extreme domain reweighting and stabilizes training.The authors report that too high a temperature can nevertheless slow convergence for initializations far from the Pareto front.Source: Figure 17 · p. 33, §A.7 · p. 33
• Given RL training’s computational cost, the authors use T = 5 and α = 0.5 in the RLVR-only setting and retain them for the hybrid setup without further large-scale tuning.Table 5 varies α, T, and window size W.Source: §A.7 · p. 33, Table 5 · p. 33
• α = 0.25 raises SAT by 0.1% and ScienceQA by 0.6%, but lowers ChartQA and InfoVQA by 3%.Up-weighting the instability term can increase the accuracy reward’s weight because it fluctuates substantially, at the cost of perception skills.Source: §A.7 · p. 33, Table 5 · p. 33
• Lowering temperature yields the highest MathVista score but reduces SAT performance by 3.2%.The authors note that decreasing temperature induces greater variation across tasks.Source: §A.7 · p. 33, Table 5 · p. 33