PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training

Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi, Qingfu Zhang, Hongbin Liu, Gaofeng Meng, Fei Zhu

arXiv:2507.05386v6cs.LGcs.AIcs.CL

TL;DR

Continual post-training must adapt multimodal models to evolving tasks while retaining prior knowledge, but the role of the learning paradigm remains underexplored. The paper compares SFT and RFT across multimodal tasks and finds that RFT preserves downstream and general capabilities, while selective updates motivate RIF-RFT for more efficient training.

  • Problem

    The role of SFT versus RFT in preserving knowledge during continual post-training remains underexplored, despite catastrophic forgetting being a central CPT challenge.

    Source: §1 · p. 1

  • Method

    The study compares SFT and RFT on sequential multimodal tasks using Qwen2.5-VL-7B-Instruct, analyzes RFT learning dynamics, and proposes rollout-based instance filtering.

    Source: §1 · pp. 1–2

  • Results

    RFT preserves prior task performance and general capabilities, whereas SFT causes catastrophic forgetting and severe general-capability degradation; RFT reaches performance comparable to multi-task training.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    RFT is a robust continual post-training paradigm whose selective updates protect established knowledge, while RIF-RFT improves its stability and sample efficiency.

    Source: §1 · p. 2, §6 · p. 9

    Results include single-run experiments due to computational cost, and rigorous theoretical characterization of RFT stability remains open.

    Source: D. Limitations and Future Work · p. 16

Abstract

from arXiv · show

Continual post-training (CPT) is a popular and effective technique for adapting foundation models like multimodal large language models to ever-evolving downstream tasks. While existing research primarily focuses on methods like data replay, model expansion, or parameter regularization, the fundamental role of the learning paradigm remains largely unexplored. This paper presents a comparative analysis of two core post-training paradigms: supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT), investigating their respective impacts on knowledge retention during CPT. Our experiments are conducted across multiple multimodal tasks, utilizing Qwen2.5-VL-7B-Instruct as the base model. The investigation yields two significant findings: (1) When continuously learning on downstream tasks, SFT leads to catastrophic forgetting of previously learned tasks. In contrast, RFT inherently preserves prior knowledge and achieves performance comparable to multi-task training. (2) RFT successfully protects and even enhances the model's general knowledge on standard benchmarks, while SFT degrades general model capabilities severely. Further analysis reveals that this stability is not primarily due to explicit mechanisms like KL penalty or chain-of-thought reasoning. We investigate RFT's learning dynamics and find that its selective update mechanism inherently prevents interference with established knowledge. Based on this insight, we propose a rollout-based instance filtering algorithm (RIF-RFT) that enhances the training efficiency of RFT by focusing on learnable samples. Our comprehensive study demonstrates the superiority of RFT as a robust paradigm for continual post-training.

1. Introduction

The paper compares SFT and RFT as continual post-training paradigms for multimodal models, finding that RFT preserves prior task knowledge and general capabilities more effectively than SFT.

1. Introduction

  • The study directly compares SFT and RFT for knowledge preservation across sequential multimodal downstream tasks and general benchmarks using Qwen2.5-VL-7B-Instruct.Evaluations include learned tasks and general benchmarks such as MMMU, MMLU-Pro, and POPE.Source: §1 · pp. 1–2
  • RFT maintains strong performance on previously learned tasks throughout sequential training, whereas SFT exhibits dramatic catastrophic forgetting.RFT achieves performance comparable to multi-task training without data replay, unlike SFT even with continual-learning strategies.Source: §1 · p. 2, Figure 1 · p. 2
  • RFT preserves and can enhance general capabilities after downstream continual training, while SFT severely degrades them.On MMMU, SFT drops performance from 52.1% to 40.1%, whereas RFT increases it from 52.1% to 54.2%.Source: §1 · p. 2
  • RFT’s forgetting mitigation is linked to selective updates that focus learning on samples within model competence rather than primarily to KL regularization or CoT reasoning.Samples beyond current capability contribute minimal harmful updates.Source: §1 · p. 2
  • RIF-RFT uses rollout-based instance filtering to improve RFT training efficiency and stability by focusing on learnable samples.The method is motivated by unhelpful samples that add noise and computational overhead.Source: §1 · p. 2

2. Related Works

Related work studies continual learning, foundation-model post-training, and strategies for mitigating forgetting in multimodal models. This paper distinguishes its contribution by comparing SFT and RFT directly and identifying inherent RFT stability without reasoning-format dependence.

2. Related Works

  • Continual learning for MLLMs aims to absorb streams of multimodal tasks without catastrophically forgetting previously acquired knowledge.Existing MLLM CPT research largely adapts forgetting-mitigation strategies within an SFT paradigm.Source: §2 · pp. 2–3
  • SFT is a common approach for adapting foundation models to task-specific or instruction-formatted downstream datasets.Prior work reports improved performance and generalization by scaling tasks and model size and incorporating CoT data.Source: §2 · p. 3
  • RFT has gained prominence for aligning models with preferences or improving specific objectives, with GRPO among the representative methods discussed.The related work positions RFT as a distinct post-training paradigm rather than merely an SFT forgetting-mitigation strategy.Source: §2 · p. 3
  • Prior findings that reasoning trajectories can reduce SFT forgetting complement this paper’s claim that RFT provides inherent mitigation without requiring a reasoning format.Together, these results inform post-training paradigm selection.Source: §2 · p. 4

3. Preliminaries

The paper frames post-training as adapting a multimodal policy to downstream tasks through SFT or RFT, then formalizes continual learning over sequential tasks. SFT maximizes likelihood, whereas RFT optimizes rewards using policy-gradient estimators and variance-reduction baselines.

SFT

  • SFT maximizes the likelihood of ground-truth responses by minimizing a negative log-likelihood objective.Source: §3 · p. 4

RFT

  • RFT treats the model as a policy that generates candidate responses and maximizes their expected reward.Source: §3 · p. 4
  • RFT gradients are estimated with policy-gradient methods, including REINFORCE, whose high variance motivates more stable advantage estimators and baselines.Source: §3 · p. 4
  • GRPO uses group-normalized rewards to form response advantages and can include a KL-divergence penalty against a reference policy.Source: §3 · p. 4
  • RLOO reduces variance by using the average reward of the other sampled responses as the baseline for each response.Source: §3 · p. 4

Continual Post-Training Formulation

  • Continual post-training learns sequential task datasets while optimizing each task objective under the constraint of retaining performance on previously learned tasks.Source: §3 · p. 4

4. Reinforcement Fine-Tuning Mitigates Forgetting in CPT

The study compares SFT and RFT for continual post-training and finds that RFT preserves prior task knowledge and general capabilities more effectively, including against replay-based baselines.

Main comparison

  • Sequential SFT produces catastrophic forgetting, with FM of -10.4%, ScienceQA falling from 95.2% to 76.1%, and final AvgAcc reaching only 54.0%.Multi-task SFT reaches 62.9%, establishing a substantially higher upper bound.Source: §4.2 · p. 5, Table 1 · p. 6
  • GRPO achieves 60.0% final AvgAcc with FM of -2.3%, approaching the 62.9% multi-task upper bound while retaining prior task performance.ScienceQA remains at 93.0% versus a 95.6% peak after the full sequence.Source: §4.2 · pp. 5–6, Table 1 · p. 6

Comparison with replay

  • RFT outperforms experience replay, reaching FM of -0.4% versus -2.8% for replay-enhanced SFT without external memory buffers.The comparison uses a 25% replay ratio; replay also introduces storage overhead and potential negative transfer.Source: §4.2 · p. 6

Robustness

  • Across model scales and task domains, RFT maintains near-zero forgetting while SFT suffers substantially larger performance drops.The analysis evaluates Qwen2.5-VL-3B-Instruct, Qwen3-VL-8B-Instruct, and a text-only Qwen2.5-7B-Instruct model.Source: §4.2 · p. 6

General capabilities

  • RFT preserves general capabilities, with GRPO improving MMMU by 2.1% and POPE by 1.9%, whereas SFT lowers MMLU-Pro by 16.9%.The reported GRPO changes are 52.1% →54.2% on MMMU and 86.6% →88.5% on POPE.Source: Table 2 · p. 6, §4.3 · p. 7

5. Understanding RFT’s Forgetting Mitigation

Ablations and sample-level analyses indicate that RFT’s forgetting resistance does not primarily depend on KL regularization or CoT, but on selective updates driven by informative learning signals. RIF-RFT operationalizes this insight by filtering unproductive samples to reduce computation while retaining stronger continual-learning performance than SFT.

Ablation studies

  • Removing KL or CoT does not eliminate RFT’s resistance to forgetting, indicating these mechanisms are not its primary forgetting-mitigation cause.Without KL, training becomes less stable later; without CoT, the model still resists forgetting and can outperform standard GRPO on some evaluations.Source: §5.1 · pp. 7–8

Selective updates

  • RFT selectively updates parameters according to learning-signal informativeness, unlike SFT’s unconditional push toward ground-truth responses.In GRPO, identical rewards collapse the advantage and produce no update when behavior is already stable, whereas SFT continues computing gradients.Source: §5.2 · p. 8

Sample difficulty

  • Easy samples enable stable learning, while hard samples can degrade training because absent positive rewards make their learning signal noisy.Mixtures perform comparably to easy-sample training, supporting informative samples as the driver of effective learning.Source: §5.2 · p. 8, Figure 4 · p. 8, §5.3 · p. 8

RIF-RFT

  • RIF-RFT filters each task’s instances using rollout rewards before standard RFT, retaining samples whose rollouts exceed a minimal threshold τ.The method trades away some adaptability because initially unrewarded samples may later become learnable.Source: §5.3 · p. 9
  • Using 50% of the data, RIF-RFT achieves 57.5% AvgAcc versus 54.0% for SFT and substantially outperforms SFT in forgetting mitigation.Full-data GRPO remains best overall, so RIF-RFT trades some performance for computational savings.Source: §5.3 · p. 9, Table 9 · p. 9

6. Conclusion

The paper concludes that the learning paradigm itself is central to continual post-training stability: RFT naturally mitigates catastrophic forgetting where SFT severely degrades task-specific and general capabilities. It also introduces RIF-RFT to improve RFT’s stability and sample efficiency.

6. Conclusion

  • RFT naturally preserves task-specific and general knowledge during continual post-training, while SFT causes severe degradation.The conclusion presents RIF-RFT as improving RFT’s stability and sample efficiency without compromising robustness.Source: §6 · p. 9

Impact Statement

The impact statement frames continual post-training as a retention problem with deployment implications, while emphasizing that preservation must be paired with safety evaluation and efficient training.

Impact Statement

  • More stable post-training may reduce repeated-retraining costs and improve deployed-system reliability, but retaining model behavior can also preserve unsafe capabilities or biases.The statement recommends careful evaluation, safety filtering, and targeted unlearning when necessary.Source: Impact Statement · p. 9
  • RFT remains computationally expensive, motivating continued work on efficient protocols and transparent compute reporting.Source: Impact Statement · p. 9

A. Dataset Information

The study uses diverse multimodal datasets for continual post-training and evaluates retention, general knowledge, and specialized multimodal abilities with benchmark suites and additional analyses.

Multimodal datasets

  • The dataset suite spans multimodal training and evaluation resources covering science, OCR, accessibility, compositional reasoning, geometry, medical images, and synthetic relational reasoning.Examples include ScienceQA, TextVQA, VizWiz, GQA, Geometry3K, PathVQA, and Super-CLEVR.Source: A. Dataset Information · p. 13

General knowledge benchmarks

  • General-knowledge evaluation uses MMMU, MMLU-Pro, and POPE to assess multimodal knowledge, challenging reasoning, and object hallucination.Source: A. Dataset Information · p. 14

Additional analyses

  • The appendix reports additional analyses of computational efficiency and filtering thresholds, while generalization across model scales, text-only tasks, and task orderings is reported in the main paper.RIF-RFT’s filtering overhead is under 2% of total time and reduces total time by approximately 44%.Source: B. Additional Analysis · p. 14, §B.1 · p. 14, Table 10 · p. 14

B.2. Ablation on Filtering Threshold in RIF-RFT

The ablation shows that RIF-RFT performs best with the default threshold τ = 0, because retaining low but non-zero-reward samples preserves informative training signals while higher thresholds reduce data volume.

Threshold ablation

  • The default threshold τ = 0 achieves the highest overall performance, retaining samples that receive any non-zero reward across rollouts.Higher thresholds retain fewer samples and reduce training data volume.Source: §B.2 · p. 14, Table 11 · p. 14
  • As τ increases, performance degrades across all tasks because progressively fewer training instances are retained.Source: §B.2 · p. 14

Random-matched comparison

  • RIF-RFT outperforms a random-matched GRPO baseline on the PathVQA →sCLEVR subsequence, indicating that gains come from selecting informative samples rather than merely reducing data.Source: §B.3 · p. 15, Table 12 · p. 15

B.4. POPE Split-Wise Results

Split-wise POPE results show that SFT harms hallucination-related performance most on adversarial and random splits, whereas GRPO improves over the base model across all three splits. The broader comparison frames this stability as a CPT-specific effect linked to selective learning on samples within current model competence.

POPE split-wise results

  • SFT degrades POPE performance most severely on the adversarial and random splits, while GRPO improves over the base model across all three splits.POPE includes random, popular, and adversarial splits, with adversarial often the most discriminative for object hallucination.Source: §B.4 · p. 15, Table 13 · p. 15

Discussion

  • The study addresses sequential multi-task CPT for MLLMs, unlike concurrent work focused mainly on single-task adaptation, and explains stability through gradient dynamics and sample difficulty.Source: §C.1 · p. 15, Table 14 · p. 15, §C.2 · p. 16
  • The analysis finds that removing the explicit KL penalty does not produce catastrophic forgetting in the CPT setting, so KL regularization is not the primary driver of RFT’s stability.Source: §C.2 · p. 15
  • RFT’s selective updates focus learning on samples within current competence, while SFT updates on all samples, allowing hard-sample noise to accumulate across sequential tasks.Source: §C.2 · pp. 15–16
  • The paper proposes a practical filtering algorithm that improves RFT efficiency for continual-learning pipelines.Source: §C.2 · p. 16

D. Limitations and Future Work.

The study’s conclusions are limited by single-run evaluation, incomplete theoretical understanding, and comparison against only a subset of continual post-training methods.

Limitations and Future Work

Related papers

Loading 2507.05386v6…