PaperlayerView exact revision on arXiv

Source-linked AI summary

Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning

Meng Lou, Hanzhong Guo, Linwei Chen, Yizhou Yu

arXiv:2605.09640v1cs.CVcs.LG

TL;DR

Whether reinforcement fine-tuning can overcome catastrophic forgetting in challenging visual continual learning remains unclear. This paper proposes RaPO, which shapes trajectory-level rewards and normalizes advantages across tasks, achieving leading performance while reducing forgetting and preserving plasticity.

  • Problem

    Whether reinforcement fine-tuning can effectively overcome catastrophic forgetting in challenging visual continual learning, including class- and domain-incremental learning, remains open.

  • Method

    RaPO uses a Retention Reward to reinforce low-drift rollouts and Cross-Task Advantage Normalization to stabilize optimization across task boundaries.

  • Results

    RaPO consistently achieves leading performance across diverse visual continual learning settings while substantially reducing catastrophic forgetting and preserving strong plasticity.

  • Takeaways & Limitations

    The study provides a systematic exploration of reinforcement fine-tuning for visual continual learning and identifies trajectory-level drift as a forgetting-related bottleneck.

  • Takeaways & Limitations

    The study evaluates Qwen2-VL at 2B and 7B scales in closed-world settings with explicit task boundaries and verifiable rewards.

Abstract

from arXiv · show

Recent studies suggest that Reinforcement Fine-Tuning (RFT) is inherently more resilient to catastrophic forgetting than Supervised Fine-Tuning (SFT). However, whether RFT (e.g., GRPO) can effectively overcome forgetting in challenging visual continual learning settings, such as class-incremental learning (CIL) and domain-incremental learning (DIL), remains an open problem. Through a pilot study, we confirm that while RFT consistently outperforms SFT, it still suffers from non-negligible forgetting. We empirically trace this bottleneck to Trajectory-level Drift Agnosticism: among candidate rollouts achieving identical task rewards, the KL divergence from the preceding-task policy varies substantially, which strongly correlates with catastrophic forgetting across sequential tasks. Motivated by this insight, we propose Retention-aware Policy Optimization (RaPO), a simple yet effective RFT method that explicitly mitigates forgetting through trajectory-level reward shaping. Specifically, RaPO comprises two core components: (1) Retention Reward that converts trajectory-level distribution drift into a continuous reward signal, preferentially reinforcing knowledge-preserving rollouts within each group; (2) Cross-Task Advantage Normalization (CTAN), which maintains a persistent exponential moving average of reward statistics across task boundaries to stabilize the optimization progress during continual learning. Leveraging the free-form textual generalization of MLLMs, we comprehensively evaluate RaPO across five visual continual learning settings. Extensive experiments demonstrate that RaPO achieves leading performance, substantially reducing catastrophic forgetting while preserving strong plasticity. To the best of our knowledge, this work represents the first systematic exploration of RFT in visual continual learning, offering insights that we hope will inspire future research.

1 Introduction

RFT is more resilient to catastrophic forgetting than SFT, yet its effectiveness in challenging visual continual learning remains open. The paper attributes residual forgetting to drift differences among equally rewarded rollouts and proposes RaPO to preserve knowledge through trajectory-level reward shaping and cross-task normalization.

  • Motivation: RFT is naturally more resilient to catastrophic forgetting than SFT, but its efficacy in challenging visual continual learning remains an open problem.The settings include class-incremental learning (CIL) and domain-incremental learning (DIL).
  • Pilot Study: In rehearsal-free few-shot ImageNet-R CIL, GRPO consistently outperforms SFT but still suffers from significant forgetting.The pilot study splits 200 classes into 10 non-overlapping tasks, with 20 classes and 5 labeled examples per class per task, using Qwen2-VL-2B and a joint-training upper bound.
  • Pilot Study: Among equally rewarded rollouts, token-level KL divergence from the preceding-task policy varies substantially, with the difference enlarging as tasks progress.The analysis isolates drift differences by comparing rollout groups with maximal task reward.
  • Method: RaPO mitigates catastrophic forgetting through trajectory-level reward shaping that converts drift from the preceding-task policy into a dense retention signal.The Retention Reward explicitly reinforces knowledge-preserving rollouts when task rewards are comparable.
  • Method: Cross-Task Advantage Normalization maintains a persistent smoother of reward scale to prevent abrupt advantage fluctuations at task boundaries.Together with Retention Reward, CTAN supports adaptation to new data while remaining anchored to previously acquired knowledge.
  • Evaluation: RaPO is evaluated across five visual continual learning settings spanning incremental image classification, object detection, and video classification.The settings include class- and domain-incremental image classification, class- and domain-incremental object detection, and class-incremental video classification.

2 Related Work

Related work establishes RFT as a reasoning-oriented paradigm increasingly centered on verifiable rewards, while visual continual learning targets adaptation to non-stationary streams without catastrophic forgetting. This work positions RaPO as the first systematic exploration of RFT for challenging visual continual learning and reports leading performance with strong generalizability.

  • Reinforcement Fine-Tuning: RFT has demonstrated strong capacity to incentivize reasoning in LLMs, with research increasingly shifting from human feedback toward verifiable rewards.GRPO has helped motivate this growing research direction.
  • Visual Continual Learning: Visual continual learning adapts models to non-stationary visual streams while seeking to avoid catastrophic forgetting.Rehearsal-free CIL is described as a particularly representative and challenging paradigm because it incrementally introduces classes without historical training data.
  • This Work: RaPO explores RFT in challenging visual continual learning and achieves leading performance with strong generalizability across different baselines.The authors identify this as the first systematic exploration of RFT’s potential in visual continual learning.

3 Method

RaPO addresses rehearsal-free visual continual learning by adding a trajectory-level retention reward to GRPO and stabilizing advantage normalization across task boundaries. Its design preserves anchor-proximal rollouts while maintaining controlled task adaptation and a persistent optimization scale.

  • GRPO backbone: GRPO samples rollout groups from the current policy, assigns task-specific verifiable rewards, and computes group-relative advantages for clipped policy optimization.For multimodal inputs, rewards include accuracy for image/video classification and IoU for object detection.
  • Continual-learning setup: Rehearsal-free CIL and DIL train each arriving task using only its current data, initializing the actor from the previous task while retaining a frozen anchor copy.The learner has no access to historical training data from earlier tasks.
  • Retention Reward: RaPO converts trajectory-level drift from the previous-task anchor into a bounded exponential retention reward, assigning higher rewards to rollouts that remain closer to the anchor.The drift estimate averages per-token log-probability ratios over rollout length, and the retention reward approaches 1.0 for anchor-proximal trajectories.
  • Retention Reward: The additive retention term enters before group-relative advantage computation, re-ranking similarly task-rewarded candidates while bounding its influence by λ.Within similar-Rtask groups, anchor-closer rollouts receive stronger reinforcement; λ > 0 balances adaptation and retention.
  • Cross-Task Advantage Normalization: CTAN replaces per-batch reward dispersion with a persistent EMA normalizer, preserving within-group ranking while stabilizing advantage magnitudes across optimization steps and task boundaries.The EMA state is saved at the end of one task and loaded at the next; β can be 0.99.

4 Experiments

RaPO is evaluated across five visual continual-learning settings, including image, video, and object-detection tasks under class- and domain-incremental formulations. It consistently improves accuracy and reduces forgetting against GRPO across the reported benchmarks.

  • Experimental scope: Experiments cover class-incremental image and video classification, class-incremental object detection, and domain-incremental image classification and object detection.The default RaPO hyperparameters are n = 8, α = 20, λ = 0.5, and β = [passage truncated].
  • Class-incremental image classification: Class-incremental image classification uses ImageNet-R, ImageNet-A, TinyImageNet, and CUB-200 with 10-task/20-class and 20-task/10-class splits, averaged over three class orders.The experiments use a 5-shot protocol with five labeled samples per class at each task.
  • Class-incremental image classification: RaPO achieves the best class-incremental image-classification performance across datasets and splits, improving ImageNet-R accuracy from 74.67% to 85.92% and reducing forgetting from 20.02% to 4.69% over GRPO.On ImageNet-A, accuracy rises from 37.37% to 44.61% and forgetting drops from 28.88% to 20.16%.
  • Class-incremental object detection: On COCO 2017 class-incremental object detection, RaPO achieves the best Ab and lowest Fb in both task settings.Compared with GRPO, Ab rises from 14.64% to 19.31% and Fb falls from 6.67% to 1.39% for five tasks; for ten tasks, Ab rises from 14.30% to 19.12% and Fb falls from 6.73% to 1.37%.
  • Class-incremental video classification: RaPO leads on class-incremental video classification, attaining A of 71.79% with 10.92% F versus GRPO’s 68.12% and 16.94% on UCF-101.On Kinetics-200 with five tasks, A improves from 70.33% to 74.18% and F decreases from 30.37% to 16.76%.
  • Domain-incremental learning: Across domain-incremental classification and detection, RaPO improves performance, including DomainNet A from 64.54% to 66.27% and F from 0.31% to 0.17%.For domain-incremental object detection, RaPO reaches 37.18% Ab, 2.60% above GRPO, compared with 35.40% for joint-training SFT and 39.13% for joint-training GRPO.

5 Conclusion

The study finds that RFT is more resilient to catastrophic forgetting than SFT in visual continual learning, but still experiences non-negligible forgetting from trajectory-level drift agnosticism. It proposes RaPO, including a Retention Reward that reinforces low-drift, knowledge-preserving rollouts.

  • 5 Conclusion: RFT is more resilient to catastrophic forgetting than SFT in visual continual learning, yet still suffers from non-negligible forgetting.This residual forgetting is attributed to trajectory-level drift agnosticism.
  • 5 Conclusion: Trajectory-level drift agnosticism is identified as the cause of RFT’s remaining catastrophic forgetting.
  • 5 Conclusion: RaPO addresses catastrophic forgetting through a Retention Reward that preferentially reinforces low-drift, knowledge-preserving rollouts within each group.

A Policy-Gradient Compatibility and Optimization Stability of RaPO

RaPO’s detached retention reward is compatible with score-function policy gradients and keeps rewards, advantages, and idealized gradient estimates bounded under stated assumptions. Its stationary-point guarantee is inherited only for an idealized fixed detached surrogate, not the practical clipped, time-varying GRPO procedure.

  • Policy-gradient compatibility: The retention reward is used through one-sided drift truncation, while sampled drift values may be negative and are not themselves non-negative KL estimates.Its expectation corresponds to a token-averaged forward log-ratio contribution relative to the preceding-task anchor.
  • Policy-gradient compatibility: RaPO treats retention drift as a detached scalar return, so policy gradients remain likelihood-ratio updates without an additional pathwise gradient term.Practical clipping and finite-group normalization make the estimator approximate the ideal score-function identity.
  • Bounded reward and advantage: RaPO’s retention reward enlarges the reward range by at most λ, while CTAN lower-bounds its normalization denominator by ϵ, preventing unbounded advantages.With B = Rmax + λ, group mean rewards lie in [0, B].
  • Bounded reward and advantage: Under a bounded second moment for the trajectory score, the score-function estimator weighted by RaPO’s advantage also has a bounded second moment.This connects bounded rewards and advantages to the usual bounded-gradient condition.
  • Idealized convergence: Under idealized non-convex stochastic policy-gradient assumptions, stochastic gradient ascent on a fixed detached RaPO surrogate inherits a stationary-point guarantee.A complete convergence theorem for practical RaPO would additionally need to account for clipping bias and sample-dependent group normalization.

B More Detailed Implementations · B.1 Prompt Template and Output Format

The implementation uses task-aware closed-set prompts that expose only the cumulative vocabulary of classes introduced so far. Classification outputs are constrained to seen class names, while detection requires a JSON list whose category fields draw from the cumulative seen vocabulary.

  • B.1 Prompt Template and Output Format: At task T_t, classification introduces novel class names ΔY_t and expands the cumulative vocabulary as Y_≤t = Y_≤t−1 ∪ ΔY_t.
  • B.1 Prompt Template and Output Format: The classification prompt explicitly lists every class in Y_≤t as the candidate answer space.
  • B.1 Prompt Template and Output Format: Future-task class names are never exposed during classification evaluation.
  • B.1 Prompt Template and Output Format: The classification setup evaluates recognition and retention only for classes already introduced.
  • B.1 Prompt Template and Output Format: At task T_t, object detection introduces novel class names ΔC_t and forms the cumulative seen vocabulary C_≤t = C_≤t−1 ∪ ΔC_t.
  • B.1 Prompt Template and Output Format: Object detection requires the model to output a JSON list of detected instances.
  • B.1 Prompt Template and Output Format: Each detection entry’s "category" field is restricted to a closed-set prediction over C_≤t.
  • B.1 Prompt Template and Output Format: This detection constraint is intended to ensure evaluation reflects recognition and retention of previously learned objects.

B.2 Task Reward Function Design

The task reward functions combine task-specific correctness with strict output-format compliance. Classification verifies normalized exact-match class names, while detection parses and normalizes predicted boxes before Hungarian bipartite matching based on class agreement and IoU.

  • Classification Reward: Classification reward combines answer correctness with format compliance for image and video classification.The verifier uses an exact-match class-name reward and checks the complete <think></think> <answer></answer> structure.
  • Classification Reward: Class-name matching lower-cases predictions and targets and maps underscores, hyphens, and periods to spaces.This normalization tolerates superficial formatting variation while preserving class identity.
  • Detection Reward: Detection reward combines localization quality, class-name accuracy, and format compliance for object detection.The detector prompt asks for all objects belonging to the cumulative class vocabulary and their class names.
  • Detection Reward: The detection verifier parses JSON boxes, normalizes class names and coordinates, discards degenerate boxes, and scores same-class pairs by IoU before Hungarian assignment.Coordinates are rounded to the nearest integer and clipped to [0, 1000], while mismatched pairs receive zero score.

C Ablation Studies

The ablation studies evaluate RaPO components and hyperparameters on 10-task ImageNet-R class-incremental classification and 5-task COCO 2017 class-incremental object detection, using specified defaults unless otherwise noted.

  • Experimental Setup: Ablations cover RaPO components and hyperparameters across ImageNet-R 10-task class-incremental classification and COCO 2017 5-task class-incremental object detection.All other experimental settings follow Sections 4.1 and 4.2.
  • Default Hyperparameters: RaPO defaults to rollout group size n = 8, retention reward scale α = 20, retention reward weight λ = 0.5, and smoothing coefficient β = 0.999.These defaults correspond to Sections 3.1, 3.2.2, and 3.2.3.

C.1 Analysis of Core Components in RaPO … D.2 Impact of Different MLLMs

RaPO’s components, anchor choice, hyperparameters, KL regularization, reasoning, rollout count, policy optimizer, and model scale each affect continual-learning performance. Across these analyses, retention-aware mechanisms improve accuracy and reduce forgetting, while reasoning and sufficient rollout diversity support stable optimization.

  • C.1 Analysis of Core Components in RaPO: Adding CTAN alone raises A from 74.67 to 79.43 and Ab from 14.64 to 16.27, while reducing F from 20.02 to 13.17 and Fb from 6.67 to 4.73.Both Retention Reward and CTAN individually improve performance over vanilla GRPO on the two benchmarks.
  • C.1 Analysis of Core Components in RaPO: RaPO consistently maintains higher retention reward than diagnostic GRPO, reshaping credit assignment toward low-drift, knowledge-preserving trajectories.RaPO optimizes Rret with λ = 0.5, whereas GRPO uses λ = 0 only for measurement.
  • C.2 Effect of Anchor Policy: The immediate preceding-task policy πt−1 achieves the best results among the evaluated retention-reward anchor strategies.Alternatives include an EMA of historical policies with decay factor 0.1, the first-task policy π1, and the unfine-tuned base model πp.
  • C.3 Hyperparameter Sensitivity: Varying λ from 0.2 to 1.0, α from 10 to 40, and β from 0.9 to 0.9999 produces comparable results, with ImageNet-R last accuracy staying within a ~1.7% range.Each hyperparameter is varied individually while the other two remain fixed.
  • C.4 Impact of Standard Loss-level KL Regularization: Removing the standard loss-level KL term does not cause a large performance drop for either GRPO or RaPO.The passage indicates that loss-level KL is not the primary source of RFT’s stronger resistance to catastrophic forgetting.
  • C.5 Effect of Reasoning and Number of Rollouts: Removing intermediate reasoning degrades both accuracy and forgetting for GRPO and RaPO, while increasing rollouts from 4 to 8 steadily improves both methods.Beyond 8 rollouts, gains become marginal; for RaPO on ImageNet-R, 8 to 10 rollouts changes A from 85.92 to 86.07 and F from 4.69 to 4.53.
  • D.1 Impact of Different Policy Optimization Methods: Equipping SAPO with Retention Reward and CTAN raises accuracy to 84.46% on classification and 18.43% on detection, while reducing forgetting to 6.52% and 2.07%.SAPO alone reports A of 76.83% and F of 18.07% on ImageNet-R, with Ab of 15.28% and Fb of 6.13% on COCO.
  • D.2 Impact of Different MLLMs: Both GRPO and RaPO gain accuracy with stronger Qwen2.5-VL base models, while forgetting also increases slightly because higher early-task accuracy raises the historical performance ceiling.The evaluated models are Qwen2.5-VL 3B for classification and 7B for detection.

D.3 Qualitative Results

Qualitative examples show that both GRPO and RaPO initially perform correctly, but RaPO better retains earlier visual knowledge after the full continual-learning sequence. This pattern appears in both class-incremental classification and object detection.

  • Class-incremental image classification: In class-incremental image classification across four datasets, both methods initially recognize the target class, but GRPO often changes its final answer after later tasks.The examples use ImageNet-R, ImageNet-A, CUB-200-2011, and TinyImageNet, comparing responses on the same validation image before and after the sequence.
  • Class-incremental object detection: In COCO class-incremental object detection, both methods initially localize the bicycle, while GRPO later detects categories introduced by subsequent tasks.The protocol compares responses on the same validation image after the initial category-introducing task and after the final task.
  • Class-incremental object detection: RaPO also recognizes later categories while preserving earlier knowledge more strongly throughout continual learning.This qualitative retention advantage is shown for later categories including traffic light, person, and car.

E Limitations · F Impact Statement

The study identifies computational and scope limitations for RaPO, including expensive RFT-based MLLM training and evaluation restricted to selected models and task settings. It also reports that the work uses public resources, is domain-agnostic, and does not target harmful applications.

  • E Limitations: RFT-based MLLM training is computationally expensive, limiting scaling to longer task sequences.The hardware budget prevented evaluation on longer task sequences.
  • E Limitations: The study evaluates RaPO only on CIL and DIL using Qwen2-VL at the 2B and 7B scales.These settings define the reported experimental scope.
  • E Limitations: Whether RaPO extends to larger MLLMs remains open.The limitation concerns generalization beyond the evaluated Qwen2-VL scales.
  • E Limitations: Whether RaPO extends to more complex task formulations also remains open.The study leaves this extension for future work.
  • F Impact Statement: All experiments use publicly available datasets and open-source pre-trained models.This resource choice ensures that no private or sensitive data is involved.
  • F Impact Statement: The research is domain-agnostic and is not tailored toward harmful applications such as surveillance or misinformation dissemination.The authors therefore report no ethical concerns.
Loading 2605.09640v1…