PaperlayerView exact revision on arXiv

Source-linked AI summary

ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging

Neha Verma, Nikhil Mehta, Shao-Chuan Wang, Naijing Zhang, Alicia Tsai, Li Wei, Lukasz Heldt, Lichan Hong, Ed Chi, Xinyang Yi

arXiv:2605.12419v1cs.CLcs.IRcs.LG

TL;DR

Fine-tuning LLMs for GenRetrieval can rapidly erode general language reasoning, limiting their ability to retain both specialized and general capabilities. ORBIT tracks distance from the original model and applies distance-triggered weight averaging, outperforming existing regularization and averaging methods across recommendation and text benchmarks.

  • Problem

    Fine-tuning LLMs for GenRetrieval causes catastrophic forgetting that degrades pre-existing general-purpose capabilities.

  • Method

    ORBIT tracks inter-model parameter distance and applies weight averaging when the fine-tuned model exceeds a threshold from its original parameters.

  • Results

    ORBIT outperforms existing regularization and averaging-based methods across GenRetrieval datasets and multiple text benchmarks.

  • Takeaways & Limitations

    ORBIT preserves meaningful recommendation and text-based reasoning capabilities in LLMs adapted for GenRetrieval.

  • Takeaways & Limitations

    Repeated merges may be required theoretically to restore the distance guarantee, although they were never needed empirically in the reported experiments.

Abstract

from arXiv · show

Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of the Generative Retrieval (GenRetrieval) task. During GenRetrieval fine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses a weight averaging strategy to constrain model drift during GenRetrieval fine-tuning when this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both common continual learning baselines and related regularization methods that also employ weight averaging.

1 Introduction

Fine-tuning LLMs for GenRetrieval introduces catastrophic forgetting, with general text-based reasoning lost early and rapidly. ORBIT addresses this by tracking parameter distance and applying weight averaging during fine-tuning, outperforming existing regularization techniques across retrieval and language benchmarks.

  • GenRetrieval tokenizes items or queries as ID sequences for autoregressive prediction, but specialized LLM fine-tuning introduces catastrophic forgetting.
  • General LLM text-based reasoning performance is lost early and rapidly during GenRetrieval fine-tuning.
  • ORBIT is Pareto-dominant in recommendation performance and several language-based benchmarks versus regularization techniques, including merging-based methods.
  • ORBIT tracks inter-model parameter distance and applies weight averaging to constrain drift from the original model.Weight averaging is triggered when inter-model distance exceeds a threshold ϵ.
  • ORBIT outperforms existing regularization techniques across multiple GenRetrieval datasets and text benchmarks.
  • ORBIT uses a distinct averaging schedule from fixed-length repeated merging, reflecting flexibility and adaptability to different learning behaviors.

2 Related Work

Related work uses model merging to mitigate fine-tuning forgetting and to support continual learning. Approaches include post-hoc interpolation or recombination with pretrained parameters, as well as merging after successive task fine-tuning stages.

  • Model merging combines capabilities from multiple models by directly combining their parameters in weight space.
  • Wortsman et al. (2022b) reduce forgetting after fine-tuning by post-hoc interpolating pretrained and fine-tuned models.
  • LiNeS similarly applies post-hoc recombination of the fine-tuning task vector with pretrained parameters.
  • Continual-learning methods merge models after each task fine-tuning stage in domain-incremental learning settings.

3 Background and Motivation

This section defines GenRetrieval as next-item autoregressive generation over user histories and adapts pretrained LLMs by adding Semantic ID vocabulary parameters. It shows that GenRetrieval fine-tuning causes severe, rapid loss of original language capabilities, motivating mitigation methods suited to rapid forgetting.

  • GenRetrieval task: GenRetrieval frames sequential recommendation as autoregressive generation, using a user’s item history as context to predict the next item.Prior work represents items with unstructured, naively structured, semantically motivated, or learned token-based IDs.
  • GenRetrieval task: The approach adapts pretrained LLMs by appending Semantic ID tokens to their input and output vocabulary projections and fine-tuning the resulting parameters.This differs from encoder-decoder-based GenRetrieval learning in prior work.
  • Quantifying the forgetting problem in GenRetrieval: After GenRetrieval fine-tuning, all evaluated text benchmarks fall to levels resembling random chance or majority-class performance.BBH, Drop, and TriviaQA drop to 0, while ARC-C and BoolQ reach majority-class performance.
  • Quantifying the forgetting problem in GenRetrieval: Within the first 2000 fine-tuning steps, essentially all text performance is lost, with a benchmark average of ∼0.15 indicating full text-performance loss.The rapidity and severity of forgetting require mitigation methods designed for this regime.
  • One-round merging fails to generalize: Post-hoc interpolation between final GenRetrieval and pretrained parameters is evaluated across interpolation ratios λ, where λ = 0 denotes the initial LLM and λ = 1 denotes the GenRetrieval model.The experiment tests whether merging weights can reintroduce general capabilities and improve robustness.

4 ORBIT

ORBIT regulates GenRetrieval fine-tuning by monitoring distance from the initial model and averaging parameters when updates exceed a maximum threshold. This distance-based intervention is motivated by the observed link between model drift and rapid text-performance degradation, treating inter-model distance as a lightweight proxy for forgetting.

  • Distance measures: ORBIT uses L2-distance and Sign Dissimilarity (SD) to measure distance between initial and current model parameters.SD captures the fraction of parameters whose signs have meaningfully changed and can be computed efficiently via bitwise XOR.
  • Method: ORBIT averages the initial and current parameters after an update exceeds a fixed maximum inter-model distance.Averaging is scheduled by distance rather than a fixed training cadence.
  • Guarantee: If SD averaging fails to reduce distance below ϵ, ORBIT re-triggers merging at the next post-update check, preserving its within-distance guarantee.A sign flip survives averaging when |θcurrent(i)| > |θinit(i)|; repeated merges were never needed empirically.
  • Motivation: Checkpoint experiments found a correlation between text performance and distance from the initial model, supporting distance-triggered intervention.The observation links substantial drift from initialization with degraded text capability.
  • Motivation: Fine-tuning that moves the model far from its starting point can induce severe forgetting, motivating ORBIT’s distance constraint.Inter-model distance serves as a lightweight, gradient-free, and data-free proxy to forgetting.

5 Experimental Setup

The experiments fine-tune instruction-tuned Gemma3 on preprocessed Amazon Product Reviews data for GenRetrieval. They evaluate retrieval, sequential recommendation, and language capabilities using specified training, decoding, and benchmark procedures.

  • Datasets and Models: Instruction-tuned Gemma3 is the base model, and recommendation experiments use Amazon Product Reviews subsets Beauty, Sports and Outdoors, and Toys and Games.The instruction-tuned variant is chosen to better match the capabilities being preserved.
  • Datasets and Models: Preprocessing filters users with fewer than 5 reviews, caps histories at 20 items, and represents each item with 4 Semantic ID tokens.Each datapoint contains a user ID, previous-item SIDs, and a held-out item converted to SIDs.
  • Training and Evaluation: Baseline GenRetrieval training uses Adafactor, cosine decay with warmup, peak learning rate 0.02, 50,000 training steps, and batch size 16.The schedule specifies 10,000 warmup steps, 30,000 decay steps, and a minimum learning rate of 1e-5.
  • Training and Evaluation: Retrieval metrics use beam search with 20 beams and 20 tokens per beam, while sequential recommendation reports NDCG@10 and Recall@10.Language capabilities are evaluated with the Table 1 benchmark suite, combining scoring- and sampling-based evaluations and reporting the average.
  • Baselines: The study compares no-intervention and L2-weight-decay baselines while excluding methods requiring original training data or Fisher information.These constraints reflect the assumption that prior training data are unavailable.
  • Averaging Methods: Soup-to-Go and ORBIT use constant learning rates and fine-tune for up to 200k steps so averaging effects are not confounded by changing learning rates.The baseline uses cosine decay, but constant rates simplify comparisons between merges.

6 Results

ORBIT achieves the strongest joint text and retrieval performance by minimizing distance to the ideal point, outperforming Soup-to-Go and other baselines. Only repeated averaging methods achieve non-trivial performance in both domains under severe forgetting.

  • Joint performance: ORBIT improves joint performance beyond Soup-to-Go by minimizing distance to the ideal point.Representative checkpoints are selected from Pareto-optimal sets using DTIP and combined retention.
  • Joint performance: ∼0.3 text average and Recall@5 of ∼.02 reflect substantial text and retrieval performance relative to topline baselines.
  • Repeated averaging: Only Soup-to-Go and ORBIT, which use repeated averaging, achieve non-trivial text and retrieval performance.Weight decay and post-hoc weight averaging may fail to generalize under severe forgetting.
  • Repeated averaging: Soup-to-Go selects k = 2000, 3000, corresponding to p = 0.01, 0.015, far below the original paper’s values, all larger than 0.15.The smaller values indicate severe forgetting and greater importance of repeated averaging during GenRetrieval fine-tuning.

7 Analysis

ORBIT’s inter-model-distance-based averaging produces a learned schedule that differs from fixed-interval averaging, while retaining stronger text and recall performance than Soup-to-Go. Analysis further supports Sign Dissimilarity over L2 and shows improved overall performance when scaling to Gemma3-4B.

  • Performance comparison: All ORBIT checkpoints outperform Soup-to-Go checkpoints on text and recall performance across the Sports and Outdoors validation data and 8 text benchmarks.Figure 6 displays only Pareto-optimal checkpoints from each experiment.
  • Learned averaging schedule: ORBIT’s averaging intervals increase during training before converging to about 3000 steps, unlike Soup-to-Go’s regular fixed interval.This learned schedule reflects flexibility determined by inter-model distance.
  • Distance-metric analysis: Sign Dissimilarity is preferred to L2 distance in ORBIT, although L2 may remain suitable; experiments fix SD at 0.007 and test L2 values of 5, 50, and 500.The authors recommend SD for computational simplicity and its fractional representation of parameter change.
  • Model scaling: Evaluating ORBIT and Soup-to-Go on Gemma3-4B IT with Sports and Outdoors shows that performance scales with the larger model and provides improved overall performance.Hyperparameters for both methods were selected based on their best performance at the 1B scale.

8 Conclusion

ORBIT enables language models to support both general text and GenRetrieval functionality while reducing the loss of general language skills during specialized fine-tuning. It preserves meaningful recommendation and text-based reasoning capabilities, supporting unified conversational models.

  • Conclusion: ORBIT enables language models to perform both general text and GenRetrieval functionalities.The method is designed to support these capabilities within a single model.
  • Conclusion: ORBIT prevents LLMs from losing their general language skills during GenRetrieval fine-tuning.The method addresses capability loss caused by adaptation to the specialized task.
  • Conclusion: ORBIT preserves meaningful recommendation and text-based reasoning capabilities in LLMs adapted for GenRetrieval.These retained capabilities can help enable a unified model for conversational applications.

A Limitations

The study evaluates ORBIT only for full fine-tuning, not PEFT methods such as LoRA, and tests it on models substantially smaller than frontier systems. Whether ORBIT reduces forgetting similarly for PEFT and frontier-scale models remains untested.

  • Limitations: ORBIT is evaluated only in a full fine-tuning setting, excluding PEFT techniques such as LoRA.PEFT methods may exhibit different forgetting behaviors during fine-tuning.
  • Limitations: The experiments use 1B and 4B models, which are much smaller than frontier models.The study therefore does not establish ORBIT’s behavior at frontier scale.
  • Limitations: Whether ORBIT achieves the same forgetting reduction for PEFT or frontier-scale models remains necessary to test.The passage identifies both fine-tuning regime and model scale as open evaluation gaps.

B Amazon Product Reviews Dataset

The Amazon Product Reviews data are summarized through dataset statistics covering three subsections. These statistics are presented in Table 6.

  • B Amazon Product Reviews Dataset: Table 6 reports dataset statistics for the three subsections of the Amazon Product Reviews data.The supplied passage identifies the table and its scope but does not provide the underlying statistics.

C Soup-to-Go exploratory hyperparameters … D.4 Finite-merge recovery

The section specifies Soup-to-Go hyperparameters and establishes that repeated origin-regulated merging can eliminate initially sign-flipped coordinates after finitely many merges. It formalizes this recovery through an iterated-merge expression, a per-coordinate sign-flip condition, and a resulting zero sign-dissimilarity guarantee.

  • C Soup-to-Go exploratory hyperparameters: Table 7 reports the Soup-to-Go hyperparameters used in the distance study.
  • D.1 Setup: ORBIT merges current parameters θ0 with origin parameters θinit, producing θk = M^k(θ0) after k consecutive merges.
  • D.1 Setup: The setup assumes every coordinate counted in SD has nonzero origin parameters and excludes randomly initialized SID vocabulary entries from SD.
  • D.2 Closed form for iterated merging: Iterated merging is analyzed by induction, with the closed form established from the base case k = 0.
  • D.3 Per-coordinate sign flip: For an initially sign-flipped coordinate, recovery after k merges occurs if and only if 2^k > 1 + r_i.
  • D.3 Per-coordinate sign flip: Coordinates initially aligned with θinit remain aligned for every k ≥ 0 because averaging same-sign values cannot change their sign.
  • D.4 Finite-merge recovery: For k > log2(1 + rmax), all initially flipped coordinates recover, the contributing set becomes empty, and SD(θk, θinit) = 0.The SD-contributing set is monotonically nonincreasing in k.
Loading 2605.12419v1…