PaperlayerView exact revision on arXiv

Source-linked AI summary

SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, Yi Ma

arXiv:2501.17161v2cs.AIcs.CVcs.LG

TL;DR

The paper examines whether SFT and RL learn generalizable knowledge or memorize training data across textual-rule and visual variants. It evaluates both methods on GeneralPoints and V-IRL, finding that RL generalizes across these variants while SFT generally struggles out of distribution. The study also identifies SFT’s role in stabilizing effective RL training and reports limitations for extreme checkpoints and controlled environments.

  • Problem

    The paper asks whether SFT and RL primarily memorize training data or acquire generalizable rules that adapt to novel task variants.

  • Method

    The study compares SFT and outcome-based multi-turn RL on GeneralPoints arithmetic reasoning and V-IRL visual navigation across textual-rule and visual out-of-distribution variants.

  • Results

    RL generalizes across rule-based and visual variants, while SFT tends to memorize training data and struggle out of distribution; RL improves V-IRL mini-benchmark performance by +33.8% (44.0%→77.8%).

  • Takeaways & Limitations

    RL can acquire generalizable knowledge in multimodal tasks, while SFT remains necessary to stabilize output format for effective RL training.

  • Takeaways & Limitations

    RL has limited effectiveness when applied to extremely underfit or overfit initial checkpoints, and the study remains confined to controlled research settings without real-world deployment.

Abstract

from arXiv · show

Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used post-training techniques for foundation models. However, their roles in enhancing model generalization capabilities remain unclear. This paper studies the difference between SFT and RL on generalization and memorization, focusing on text-based rule variants and visual variants. We introduce GeneralPoints, an arithmetic reasoning card game, and adopt V-IRL, a real-world navigation environment, to assess how models trained with SFT and RL generalize to unseen variants in both textual and visual domains. We show that RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants. SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution scenarios. Further analysis reveals that RL improves the model's underlying visual recognition capabilities, contributing to its enhanced generalization in the visual domain. Despite RL's superior generalization, we show that SFT remains essential for effective RL training; SFT stabilizes the model's output format, enabling subsequent RL to achieve its performance gains. These findings demonstrates the capability of RL for acquiring generalizable knowledge in complex, multi-modal tasks.

1. Introduction

The paper asks whether SFT and RL memorize training examples or learn transferable rules, evaluating this distinction across textual and visual task variants. It reports that RL generalizes across unseen rules and visual inputs, while SFT tends to memorize and struggle out of distribution.

  • Motivation: The study separates memorization from transferable rule acquisition when comparing SFT and RL generalization.Memorization means generating near-exact copies of training examples based on training-set information, excluding bitwise or codewise replication.
  • Evaluation focus: Generalization is evaluated across textual rule variants and visual-input variations in language and vision-language tasks.The study considers both applying learned rules to altered textual rules and maintaining performance under changed visual inputs.
  • Approach: The multi-step RL framework is applied after SFT using a sequential revision formulation.The introduction identifies SFT as preceding RL in the adopted training setup.
  • Main finding: RL learns generalizable rules across GeneralPoints and V-IRL, whereas SFT appears to memorize training rules and fails to generalize.The reported pattern holds for both textual rule-based and visual out-of-distribution settings.
  • Main finding: RL improves V-IRL mini-benchmark performance by +33.8% (44.0%→77.8%).The paper presents this improvement as a by-product of the visual out-of-distribution generalization capability.

2. Related Works

Related work frames SFT as task and format adaptation, while RL and inference-time verification motivate the paper’s study of generalization, memorization, and visual capability.

  • Post-training: Post-training commonly uses large-scale supervised fine-tuning, reinforcement learning, or both to enhance model performance.The related work positions these methods as standard foundation-model post-training techniques.
  • Post-training: SFT adapts pretrained models to downstream tasks through task-specific, often instruction-formatted datasets.Prior work also reports improved zero-shot performance after diverse instruction tuning.
  • Post-training: LIMA characterizes SFT as a format teacher that adapts responses while leveraging pretrained language-model capabilities.This related-work view motivates the paper’s analysis of output-format stability.
  • Memorization and generalization: Prior studies distinguish LLM memorization of training data from generalization reflected in divergence from the pretraining distribution.The literature review also discusses differing overfitting patterns across task types.
  • Inference-time compute: The paper combines inference-time verification with multi-turn RL so models can identify and correct errors.It further examines whether increasing the maximum number of verification steps affects RL generalization.
  • Visual capability: Existing VLMs show strong performance on challenging tasks but remain limited in visual perception.The review summarizes prior efforts to improve visual capability using multiple visual encoders.

3. Preliminaries

The preliminaries formulate foundation-model training as finite-horizon reinforcement learning with text or image-conditioned states, verifier-generated rewards, and sequential revision.

  • Standard RL terminology: Finite-horizon RL defines states S, actions A, rewards r, and a maximum episode length T for learning a return-maximizing policy.The policy maps states to actions and is evaluated by expected overall return.
  • LLM/VLM formulation: For language models, states are text inputs; for vision-language models, states combine text inputs with RGB images.The action space consists of output token sequences.
  • LLM/VLM formulation: A verifier evaluates the model output and produces both an outcome-based reward and textual verifier information.The verifier maps output text to a reward and verifier-text space.
  • LLM/VLM formulation: The model acts as a policy network updated with PPO in the multi-turn RL setting.The policy maps states to output token sequences.
  • Sequential revision: Sequential revision initializes the input with a system prompt and then appends prior model and verifier outputs at later steps.This history forms the next input used for iterative revision.

4. Evaluation Tasks

The evaluation uses GeneralPoints for arithmetic reasoning and V-IRL for visual navigation, with each environment supporting rule and visual variants that test out-of-distribution generalization.

  • GeneralPoints: GeneralPoints evaluates arithmetic reasoning using four cards and requires an equation reaching a target number while using every card exactly once.Cards are represented as text in GP-L or images in GP-VL.
  • GeneralPoints: GeneralPoints rule variants assign J, Q, and K either their conventional values 11, 12, and 13 or the shared value 10.These textual rules test whether models generalize arithmetic operations rather than memorize post-training data.
  • GeneralPoints: GeneralPoints visual variants change card colors between training and out-of-distribution testing.The visual challenge is recognizing card numbers independently of color.
  • V-IRL: V-IRL evaluates spatial reasoning in open-world navigation using either pure language descriptions or vision-language input.The task requires navigating to a target location from spatial instructions and visual observations.
  • V-IRL: V-IRL rule variants use either absolute orientations such as north and east or relative actions such as left and right.These alternative action spaces test spatial knowledge beyond memorized post-training outputs.
  • V-IRL: V-IRL visual generalization trains navigation in one location and evaluates it in different locations with different landmarks.The visual challenge is recognizing landmarks and relating them to linguistic instructions.

5. Results

Across GeneralPoints and V-IRL, RL improves out-of-distribution generalization across textual rule and visual variants, while SFT generally degrades OOD performance. RL also improves visual recognition, whereas SFT is needed to make RL effective when the backbone poorly follows instructions.

  • Generalization across Rules: RL improves OOD performance across unimodal and multimodal tasks, while SFT degrades performance across every evaluated task.RL gains are reported for GP-L, V-IRL-L, GP-VL, and V-IRL-VL; SFT shows decreases in all four cases.
  • Generalization in Visual Out-of-Distribution Tasks: +17.6% in GP-VL and +61.1% in V-IRL-VL show RL generalization across visual variants, while SFT decreases performance by -9.9% and -5.6%.The visual OOD evaluations use altered card suits for GeneralPoints and routes from worldwide cities for V-IRL.
  • RL Improves Visual Capabilities: RL improves visual recognition accuracy and overall performance as compute scales, whereas SFT deteriorates both in GP-VL.The analysis measures recognition of four cards from the input image alongside rule-based OOD and visual OOD performance.
  • The Role of SFT for RL Training: Without SFT initialization, all end-to-end RL runs fail to improve because the base model generates responses that prevent retrieval of task-related information and rewards.The authors attribute this failure to poor instruction following in the base model.
  • Role of Verification Iterations: Under the same computational budget, increasing verification steps improves OOD performance by +2.15% with 3 steps, +2.99% with 5 steps, and +5.99% with 10 steps.One verification step yields only a marginal +0.48% OOD improvement.

6. Conclusion, Discussion, and Limitations

The paper finds that RL learns generalizable knowledge across textual rule and visual variations, whereas SFT tends to memorize training data. It also identifies unresolved limitations involving SFT on GP-VL, visual recognition, and RL initialization.

  • Conclusion: RL exhibits superior generalization across rule and visual variations, while SFT tends to memorize training data.This pattern appears across multimodal arithmetic and spatial reasoning tasks.
  • Limitations: SFT fails to match RL’s in-distribution performance on GP-VL despite 10 additional experiments varying learning rates and tunable components.None of these experiments shows a strong increasing trend like RL.
  • Limitations: Scaling up SFT degrades visual recognition capabilities, possibly because it overfits to reasoning tokens while neglecting recognition tokens.The authors identify this explanation as a hypothesis and leave further investigation to future work.
  • Limitations: RL cannot recover out-of-distribution performance from an overly tuned SFT checkpoint and may collapse to the training rule.The findings indicate limited RL effectiveness with extremely underfit or overfit initial checkpoints.

Impact Statement

The study is intended to advance machine learning through controlled investigations of model generalization. Its environments are simulated or synthetic, with no deployment involving real-world systems or data.

  • Impact Statement: The study investigates model generalization in controlled research settings using GeneralPoints and the simulated V-IRL environment.GeneralPoints is synthetic, while V-IRL serves as a simulated proxy for real-world tasks.
  • Impact Statement: No deployment or interaction with actual real-world systems or data was involved.The methods, environments, and tasks were constructed for understanding model generalization.

A.1. Data

GeneralPoints uses configurable card-game rules, sampled card inputs, and verifier-based rewards to study arithmetic reasoning under textual and visual variants.

  • Data: GeneralPoints samples card quadruples from a 52-card poker deck, guaranteeing at least one solution equal to 24 with an expert solver.The target point is fixed at 24 for all experiments.
  • Arguments: Training uses the face-cards-equal-10 rule, while out-of-domain evaluation uses the alternative rule and requires at least one face card.This forces calculations with numbers above 10 that are absent during training.
  • Arguments: Visual distribution-shift experiments train on black suits and evaluate on red suits.The training suits are spades and clubs; evaluation uses hearts and diamonds.
  • Reward design: Episodes end after a correct equation or five verification steps, with rewards ranging from 5 for success to negative penalties for invalid outcomes.The vision-language variant adds a -1.5 penalty for failing to recognize the cards.

B.1. Data

V-IRL models navigation as route following with language instructions, street-view observations, and oracle information, using modified dynamics and rewards for RL training.

  • Data: V-IRL training uses 1000 unique routes from New York City, while visual out-of-distribution evaluation uses 18 routes across nine cities.The benchmark routes do not overlap with the training data.
  • Data: The environment provides visual-language and pure-language transition examples, with equivalent information presented in different formats.The visual-language examples add a visual recognition challenge.
  • Data: Each route contains a real-world path with language instructions, visual signals, turning points, straight roads, street views, and oracle information.Street views are 360-degree panoramas at movable points.
  • Additional Details on the Environmental Design: V-IRL simplifies the original design by removing its two-stage navigation pipeline and online queries to reduce training time and cost.The modified environment adds configurable action-space and maximum-straight-road-length arguments.
  • Additional Details on the Environmental Design: Navigation episodes terminate when the agent reaches the destination or exceeds two verification steps, with reward 1 for a correct current-coordinate action.The reward design is adapted for reinforcement learning training.

C.1. Data

The experiments use expert prompt-response pairs for main-body SFT, then test whether more diverse suboptimal trajectories improve OOD generalization. SFT still memorizes training data and shows degraded OOD performance with suboptimal trajectories.

  • Main-body SFT uses optimal single-turn prompt-response pairs without verification or revision steps.
  • Suboptimal-trajectory SFT includes errors and verification messages, aligning its format with evaluation scenarios allowing multiple verification iterations.
  • SFT still memorizes training data and has degraded OOD performance when trained on suboptimal trajectories.The result suggests memorization arises from the fundamental nature of SFT training rather than solely from the SFT data.
  • The training pipeline initializes the model with SFT before separately scaling compute for SFT and RL.

C.3. Evaluation Metric

The evaluation reports per-step accuracy and task success rate, while estimating training and inference computation for comparing SFT and RL. Additional ablations examine learning-rate choices and plot-smoothed uncertainty.

  • Evaluation Metrics: Per-step accuracy counts an action as correct when it matches the expert trajectory at that position, with verification steps treated as independent samples.
  • Evaluation Metrics: Success rate requires at least one successful verification in GeneralPoints and correct actions at every movable route point in V-IRL.
  • Computation Estimation: Training FLOPs are estimated as X_train = 6ND_train, while inference FLOPs are estimated as X_inference = 2ND_inference.N denotes model parameters and D_train denotes training tokens.
  • Computation Estimation: PPO’s iterative replay-buffer collection and optimization require additional inference computation, approximated using generation-process and average input/output-token terms.The estimated λ is 6 for GeneralPoints and 5.1 for V-IRL.
  • Plotting and Error Bars: Line plots use a third-order Savitzky–Golay filter, and evaluated data points are modeled with binomial-distribution error estimates.
  • Ablations: Learning-rate ablations evaluate in-distribution success curves for SFT and RL under their respective hyperparameter searches.SFT searches more learning-rate settings than the two values searched for RL.

D.2. More results on V-IRL-VL

Additional V-IRL-VL results show that overall route success is far more demanding than per-step accuracy, while failure cases expose dependencies on SFT initialization and checkpoint quality. Under these conditions, neither method achieves reasonable OOD overall performance.

  • Overall Success Rate: Both training methods achieve V-IRL-VL overall success rates no higher than 1%.Overall success aggregates per-step errors and is therefore substantially more demanding.
  • Overall Success Rate: A random policy with 10% per-step accuracy would achieve approximately 10^-8% success on routes averaging 10 steps.
  • Failure Cases: RL fails without SFT initialization because the model generates unstructured responses and fails to follow the task effectively.
  • Failure Cases: RL cannot recover OOD performance from an extremely overfitted checkpoint with initial per-step accuracy below 1%.The model fails to adjust to the new rule from that checkpoint.
  • OOD Performance: Neither training method achieves reasonable out-of-distribution performance on V-IRL-VL under rule variants.
  • Failure Cases: Scaling supervised fine-tuning more frequently produces failed V-IRL transitions attributed to overfitting.
Loading 2501.17161v2…