PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models

Rongguang Ye, Zhan Zhuang, Yichen Wu, Ming Tang, Kede Ma

arXiv:2609.32499v1cs.CLcs.AI

TL;DR

Continual LLM adaptation must acquire new task knowledge while preserving prior capabilities, yet historical prompts can become less compatible as model parameters evolve and their cross-task relationships are not explicitly organized. LAPS aligns historical prompts and learns an anchored prompt space, achieving the highest average performance among compared methods while reducing forgetting on TRACE across three Qwen3 scales.

  • Problem

    Joint prompt–parameter adaptation can leave historical prompts less compatible with evolving model parameters, while transferable relationships among task prompts are not explicitly organized.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    LAPS self-distills historical prompts under the updated model, learns a Bézier-shaped anchored prompt space, and selects task-specific prompts through validation-guided search.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 6

  • Results

    On TRACE across three Qwen3 scales, LAPS achieves the highest average performance among compared methods while reducing forgetting.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results suggest that organizing task-specific prompts into a structured, searchable space can facilitate cross-task knowledge transfer during continual adaptation.

    Source: §5 · p. 10

    LAPS assumes known task identities and adds control prompts as tasks accumulate, making open-ended learning and prompt growth challenging.

    Source: §3.1 · p. 3, §5 · p. 10

Abstract

from arXiv · show

Continually adapting large language models requires acquiring new knowledge while preserving previously learned capabilities. Jointly adapting model parameters and task-specific soft prompts offers a promising solution, but faces two key limitations: historical prompts may become less effective as the model evolves, while their transferable cross-task relationships are not explicitly learned. We propose Learning an Anchored Prompt Space (LAPS), which preserves historical prompt effectiveness and learns relationships among task-specific soft prompts to facilitate positive transfer. LAPS first aligns historical prompts with the updated model through self-distillation. LAPS then constructs an anchored prompt space whose vertices correspond to learned task-specific soft prompts and whose intermediate geometry is shaped by learnable B\'ezier control prompts. Once this anchored prompt space is learned, LAPS identifies the best-performing prompt for each task on its validation set, allowing the optimized prompt to draw on knowledge acquired from observed tasks. Experiments on the TRACE benchmark across three Qwen3 model scales show that LAPS consistently outperforms distillation-based, prompt-based, and joint prompt--parameter adaptation baselines, improving average performance while reducing forgetting.

1 Introduction

Continual LLM adaptation must acquire task-specific knowledge while retaining prior capabilities, but parameter updates can interfere with prior knowledge and prompts can lose compatibility as the model evolves. LAPS aligns historical prompts and learns an anchored prompt space to support retention and cross-task transfer.

1 Introduction

  • Continual LLM adaptation must incorporate evidence from limited task data while retaining capabilities acquired from previous tasks.Source: §1 · p. 1
  • Jointly updating parameters and prompts can destabilize adaptation because historical prompts may no longer induce their original behaviors under evolved parameters.Source: §1 · pp. 1–2
  • Parameter updates offer capacity to acquire task knowledge but can interfere with prior knowledge, whereas prompt-only methods preserve task behaviors but limit adaptation capacity.Source: §1 · p. 1
  • LAPS aligns historical prompts and learns an anchored prompt space to preserve prior adaptations while identifying prompts that exploit positive transfer.Historical prompts are aligned through on-policy self-distillation; task-specific prompts anchor a degree-k Bézier simplex whose geometry is shaped by learnable control prompts.Source: §1 · p. 2
  • LAPS learns control prompts from cross-task interactions, then selects task-specific simplex coordinates by validation performance to discover positive transfer.Source: §1 · p. 2
  • On TRACE, LAPS achieves a favorable adaptation–retention trade-off across Qwen3 models from 1.7B to 8B against distillation-based, prompt-based, and joint adaptation methods.Source: §1 · p. 2

2 Related Work

Related work stores and reuses prior knowledge through replay, distillation, prompts, or joint prompt–parameter updates. LAPS targets evolving models by aligning historical prompts and learning a controlled space for cross-task transfer.

Replay-based Continual Learning

  • Replay methods retain or regenerate historical information, but mixing replay data directly into updates may interfere with current-task adaptation.Examples include stored samples, historical logits, generative replay, and on-policy replay for LLM adaptation.Source: §2 · p. 2

Knowledge Distillation-based Continual Learning

  • Distillation preserves selected earlier behaviors, but behavioral constraints can limit acquisition of new-task knowledge; LAPS instead distills historical prompts without constraining backbone updates.Prior methods match output distributions or soft predictions and intermediate representations.Source: §2 · pp. 2–3

Prompt-based Continual Learning

  • Prompt-only methods store task knowledge in prompts and reduce forgetting, while a fixed backbone limits new-task adaptation through parameter updates.Approaches include task-aware prompt retrieval, shared and task-specific prompts, prompt concatenation, and attention-based prompt composition.Source: §2 · p. 3

Joint Prompt and Parameter Adaptation

  • Joint prompt–parameter methods face historical prompt incompatibility as model parameters evolve, and prior approaches do not explicitly align prompts or learn a controlled transfer space.LAPS aligns historical prompts under the current model, learns Bézier control prompts, and selects simplex coordinates by validation performance.Source: §2 · p. 3

3 Learning an Anchored Prompt Space for Continual Adaptation

LAPS addresses model drift and unorganized cross-task prompt relationships by aligning historical prompts, learning a replay-shaped anchored prompt space, and selecting task prompts using validation performance.

3.1 Problem Setup and Overview

  • Tasks arrive sequentially without access to earlier full training sets, and the known task identity at inference selects its task-specific prompt.The setting evaluates the model on all observed tasks after each learning stage.Source: §3.1 · p. 3
  • At each stage, LAPS jointly adapts model parameters and the current task prompt through alternating optimization, then updates the model with that prompt fixed.Previously learned prompts are frozen in the basic joint-adaptation setting.Source: §3.1 · pp. 3–4

3.2 Historical Prompt Alignment via Self-Distillation

  • On-policy self-distillation realigns historical prompts after model updates, matching the previous model–prompt pair’s predictive behavior while keeping updated model parameters fixed.Alignment uses current-task examples alongside the stage replay set; replay-only supervised tuning is insufficient because replay is sparse and targets omit broader model behavior.Source: §3.2 · pp. 4–5

3.3 Anchored Prompt Space Learning

  • LAPS fixes aligned task prompts as simplex anchors and uses learnable non-vertex Bézier control prompts to shape intermediate prompts beyond fixed convex interpolation.With degree k=1, the space reduces to convex interpolation; higher degrees add learnable intermediate control prompts.Source: §3.3 · p. 5
  • With model parameters and task anchors fixed, LAPS trains only the control prompts using replay losses across observed tasks and sampled trade-offs.A smooth Tchebycheff scalarization aggregates the trade-off-weighted task losses and emphasizes larger weighted losses.Source: §3.3 · p. 5

3.4 Validation-Guided Prompt Selection

  • For each observed task, LAPS selects a prompt from the learned space by maximizing that task’s validation score with the final model fixed.Bayesian optimization treats the score as a black-box objective, using a Gaussian-process surrogate and Expected Improvement to propose coordinates.Source: §3.4 · p. 6

4 Experiment

On TRACE, LAPS achieves the highest Average across three Qwen3 scales while keeping BWT close to or above zero. Its experiments examine adaptation dynamics, prompt-space selection, and how its components and settings affect the adaptation–retention trade-off.

4.1 Experimental Setup

  • TRACE contains eight continual instruction-tuning tasks; experiments use Qwen3-1.7B, Qwen3-4B, and Qwen3-8B and compare parameter, replay, distillation, prompt-only, and joint-adaptation baselines.Prompt-based methods use 32 tokens unless otherwise stated; results report Average and BWT, with higher BWT indicating stronger retention.Source: §4.1 · pp. 6–7

4.2 Main Results

  • LAPS records the highest Average on all three Qwen3 scales: 60.5% on 1.7B, 66.2% on 4B, and 67.5% on 8B.Relative to ProMoT, LAPS improves Average by 0.2%–2.8% and BWT by 1.4%–3.2%, while keeping BWT close to or above zero.Source: Table 1 · p. 7, §4.2 · p. 7
  • On Qwen3-1.7B, later updates degrade earlier-task performance for SFT and for ProMoT on tasks including Py150 and NumGLUE-cm; LAPS selects task prompts after sequential adaptation ends.The selected prompt improves performance on most tasks relative to the preceding pre-search point.Source: §4.2 · pp. 7–8, Figure 2 · p. 8
  • Validation-selected prompts generally lie away from their task anchors and improve average performance over anchor prompts by 0.78% on Qwen3-1.7B, 0.48% on Qwen3-4B, and 2.70% on Qwen3-8B.The PCA visualization examines selected prompts relative to the original per-task anchors.Source: §4.2 · p. 8, Figure 3 · p. 8

4.3 Ablations and Diagnostic Analyses

  • The full LAPS model outperforms each component-removal variant; removing historical prompt alignment causes the largest BWT degradation and also lowers Average.Space learning and prompt selection have complementary roles: learning prompt geometry and choosing a task-specific prompt after training.Source: §4.3 · pp. 8–9, Table 2 · p. 9
  • Adding LAPS prompt-space learning and selection improves Average and BWT across reported scales, whereas replay alone offers limited or inconsistent benefits and can reduce performance.LAPS also improves both Average and BWT when combined with OPR-SC and OPR-RU.Source: §4.3 · pp. 9–10, Table 3 · p. 9, Table 6 · p. 9
  • Increasing replay ratio does not monotonically improve performance, and LAPS remains competitive with a much smaller replay budget.The comparison tests whether retention is mainly explained by the number of previous-task examples.Source: Figure 4 · p. 8, §4.3 · p. 9
  • Across tested prompt lengths and model scales, LAPS maintains the highest Average; increasing prompt length does not consistently improve performance.For Bézier degree, learnable control prompts improve Average and BWT up to k=3, while k=4 brings no additional gain.Source: §4.3 · pp. 9–10, Table 4 · p. 9, Table 5 · p. 9

5 Conclusion and Discussion

LAPS jointly updates prompts and model parameters, aligning historical prompts and learning a searchable space for cross-task relationships. On TRACE, it achieves the highest average performance among compared methods while reducing forgetting; its current formulation assumes known task identities and adds control prompts as tasks accumulate.

5 Conclusion and Discussion

  • On TRACE across three Qwen3 scales, LAPS achieves the highest average performance among compared methods while reducing forgetting.The authors suggest that organizing task-specific prompts into a structured, searchable space can facilitate cross-task knowledge transfer.Source: §5 · p. 10
  • LAPS jointly updates prompts and model parameters, aligns historical prompts through self-distillation, learns cross-task prompt relationships, and selects prompts through validation-guided search.The learned space is structured by task-specific soft prompts and supports task-specific prompt selection.Source: §5 · p. 10
  • The current formulation assumes known task identities and adds control prompts as tasks accumulate, complicating open-ended learning with unavailable task boundaries.The authors propose task-free input-based routing and adding or consolidating control prompts as possible future directions.Source: §5 · p. 10
  • The learned space could help identify regions associated with positive transfer or interference and provide initializations that reduce data and optimization needs for new tasks.These are proposed future uses of the anchored prompt space.Source: §5 · p. 10

A.1 Properties of the Anchored Prompt Space

The appendix analyzes optimization of the shared non-vertex Bézier control prompts while model parameters and task-specific anchors remain fixed. Under stated regularity and lower-boundedness assumptions, the objective is smooth and gradient descent is guaranteed to reach first-order stationarity, not global optimality.

A.1 Properties of the Anchored Prompt Space

  • The analysis holds model parameters and task-specific prompt anchors fixed while optimizing only the non-vertex control prompts.Task losses are assumed continuously differentiable with bounded gradient and smoothness, and the expected objective is assumed lower bounded.Source: §A.1 · p. 13
  • Regular task-loss smoothness and bounded variation in the softmax coefficients imply a Lipschitz-continuous gradient for the smooth Tchebycheff objective.Taking expectation over simplex coordinates preserves gradient Lipschitz continuity, yielding a smooth expected objective.Source: §A.1 · p. 13
  • The Bézier map is affine in non-vertex control prompts when the task anchors and simplex coordinates are fixed, enabling an explicit smoothness bound.The analysis uses the Bézier parameterization and uniform simplex sampling to bound the objective's smoothness.Source: §A.1 · pp. 13–14
  • Gradient descent on the lower-bounded, smooth objective converges to first-order stationarity, not global optimality, because the objective is generally non-convex.The guarantee applies to shared control-prompt optimization with the model parameters and task-specific anchors fixed.Source: §A.1 · pp. 15–16

A.2 Bayesian Optimization for Positive Transfer Discovery

LAPS searches independently for each task over a Bézier simplex that maps coordinates to prompts, using validation scores as black-box objectives. Bayesian optimization proposes candidate coordinates, and the best validation-scoring prompt is selected for each task.

A.2 Bayesian Optimization for Positive Transfer Discovery

  • Bayesian optimization runs independently per task, maximizing that task's validation score over simplex coordinates that map deterministically to prompts.The task-specific metric may be discrete or unsuitable for gradient-based optimization, so LAPS treats validation performance as a black-box objective.Source: §A.2 · p. 16
  • The initial design evaluates all eight task vertices, the simplex centroid, and seven Sobol-generated interior coordinates.Including every task vertex ensures the original task-specific prompts are evaluated.Source: §A.2 · p. 16
  • Each candidate is evaluated on at most 500 validation examples across three inference runs, with the empirical variance used as observation noise.A Gaussian process is fitted to validation scores after transforming simplex coordinates by their elementwise square roots.Source: §A.2 · p. 16
  • Expected Improvement proposes coordinates using a Gaussian-process surrogate, balancing predicted validation scores with uncertainty.Each iteration draws candidates across the simplex, near boundaries, and around the current best coordinate.Source: §A.2 · p. 17
  • Search retains the three highest-scoring observed coordinates, adds the target-task vertex if absent, and selects the finalist with the best mean score across eight fresh runs.Each task receives at most 100 candidate evaluations, except MeetingBank, which receives 50.Source: §A.2 · p. 17

A.3 Method Characteristics and Parameter Efficiency

LAPS adds relatively few trainable parameters compared with the corresponding backbone sizes across the three Qwen3 scales.

A.3 Method Characteristics and Parameter Efficiency

  • LAPS adds 7.3M parameters for Qwen3-1.7B, approximately 0.43% of its backbone size.Table 7 compares trainable components and parameter counts.Source: §A.3 · p. 18
  • LAPS adds 9.2M parameters for Qwen3-4B, approximately 0.23% of its backbone size.Table 7 compares trainable components and parameter counts.Source: §A.3 · p. 18
  • LAPS adds 14.8M parameters for Qwen3-8B, approximately 0.18% of its backbone size.Table 7 compares trainable components and parameter counts.Source: §A.3 · p. 18

A.4 Dataset Details

TRACE evaluation covers eight tasks with varied domains, languages, and metrics. The experiments remove samples longer than 2,048 tokens and use the benchmark’s official task-specific metrics.

A.4 Dataset Details

  • Samples exceeding 2,048 tokens are removed from the experiments.Source: §A.4 · p. 18
  • Evaluation uses TRACE’s official metrics, including Accuracy, ROUGE-L, Edit Similarity, and SARI.Edit Similarity is computed from normalized Levenshtein distance.Source: §A.4 · p. 18
  • TRACE spans diverse domains, languages, and evaluation metrics across eight tasks.The benchmark includes classification and generation tasks in Chinese, English, and German.Source: §A.4 · p. 18

A.5 Additional Experiments

The additional analyses examine LAPS’s search behavior, robustness, adaptation dynamics, and prompt-space geometry. Learnable control prompts reshape the interpolation geometry and yield larger cumulative gains from space search on both model scales.

A.5 Additional Experiments

  • Learnable control prompts reshape the fixed interpolation geometry and yield larger cumulative search gains on both model scales.Source: §A.5 · p. 18
  • The analyses include search-selected task coordinates and Bayesian optimization convergence.Source: §A.5 · p. 18
  • The analyses also examine task-order robustness, continual adaptation dynamics, and how control prompts affect space geometry.Source: §A.5 · p. 18

A.5.1 Task-Specific Coordinates Selected by BO

The additional experiments show that LAPS’s search behavior varies by task and model scale, while retaining strong adaptation and more stable historical performance across the examined settings.

A.5.1 Task-Specific Coordinates Selected by BO

  • Search coordinates stay near task vertices for MeetingBank, Py150, ScienceQA, and NumGLUE-ds, but are often more distributed for C-STANCE, NumGLUE-cm, and 20Minuten.Coordinates vary substantially across model scales, supporting task- and model-specific search.Source: §A.5.1 · p. 18, Table 9 · p. 19

A.5.2 Effect of Learnable Control Prompts on Space Geometry

  • Without learnable control prompts, search is restricted to the fixed simplex interpolation geometry spanned by task-specific prompts.Source: §A.5.2 · p. 18

A.5.3 Robustness to Task Order

  • 57.4% on Qwen3-1.7B and 62.0% on Qwen3-4B Average performance under reverse task order; BWT is −2.7% and −5.7%, respectively.LAPS has the highest Average on both scales and substantially reduces forgetting versus SFT, Replay, and ProMoT; ProgPrompt has BWT 0.0% but lower Average scores of 50.6% and 56.6%.Source: §A.5.3 · p. 19, Table 10 · p. 20

A.5.4 Convergence of Bayesian Optimization

  • Bayesian optimization’s best validation performance usually improves rapidly in its first few evaluations, then gradually stabilizes across tasks and model scales.The passage indicates effective regions can be identified with a relatively small search budget.Source: §A.5.4 · p. 19, Figure 6 · p. 21

A.5.5 Additional Continual Adaptation Dynamics

A.5.6 Qualitative Analysis of Space Search

  • On a MeetingBank example, search improves per-example ROUGE-L from 8.25 at the task vertex to 45.28 while producing a more concise summary.The vertex already contains relevant information but over-generates the transcript.Source: §A.5.6 · pp. 20, 22
  • For the FOMC example, all eight task vertices predict neutral, but the search-selected interior point predicts the correct hawkish label.The selected point remains near the FOMC vertex, with small contributions from other task directions.Source: §A.5.6 · pp. 20, 23

Related papers

Loading 2609.32499v1…