PaperlayerView exact revision on arXiv

Source-linked AI summary

Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts

Jeesu Jung, Hwan Chang, Juseon Do, Jeonghwan Choi, Jinho Choo, Sungwoo Nam, S. K. Hong, Hwanjun Song

arXiv:2609.39365v1cs.CLcs.AI

TL;DR

Continual alignment must adapt to evolving requirements without forgetting earlier behaviors, yet language instructions offer indirect control and post-training repeatedly updates model parameters. Ready2Blend learns independent requirement prompts over a frozen backbone and blends them at inference. Across two incremental settings, it reaches 93.1–98.5% of joint training, with competitive retention and less training time than post-training methods.

  • Problem

    Lifelong alignment must incorporate requirements that evolve over time while preserving previously acquired behaviors, across both changing tasks and changing preferences within a fixed task.

    Source: §1 · p. 1

  • Method

    Ready2Blend learns each requirement independently as a prompt over a frozen backbone, with composable prompts jointly controllable at inference.

    Source: §1 · p. 2

  • Results

    93.1–98.5% of joint training, with competitive backward transfer, is reached by the only frozen-backbone method matching strong post-training methods across two incremental settings.

    Source: §1 · p. 3, §5.1 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    The prompt bank supports weighted personalization and order-free composition without retraining or consolidating every requirement into the backbone.

    Source: §6 · p. 10

    Steering from an unseen requirement definition without supervision is left to future work.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Continual alignment requires LLMs to adapt to new requirements without forgetting previously acquired behaviors. Natural-language instructions are flexible and composable but offer only indirect control, whereas post-training provides stronger adaptation at the cost of repeated parameter updates. We introduce Ready2Blend, which combines the flexibility of natural language with learned alignment. AlignFormer maps each requirement to a fixed-length alignment prompt stored in a modular prompt bank, while the backbone and prior prompts remain frozen. Composability regularization transfers the semantic geometry of textual requirements into prompt space, enabling inference-time blending and reweighting. Across two practical continual alignment settings, Ready2Blend is the only frozen-backbone method that matches post-training-based alignment methods, reaching $93.1$-$98.5\%$ of a joint-training reference with competitive retention, while requiring only a few prompt tokens and up to $4.3\times$ less training time. Its modular design further enables weighted personalization and order-free composition without retraining. Code will be released upon acceptance.

1 INTRODUCTION

Continual alignment must accommodate evolving requirements while preserving earlier behaviors, but language instructions provide indirect control and repeated post-training updates are costly. Ready2Blend learns requirement-specific prompts over a frozen backbone and blends them at inference.

Motivation

  • Natural-language instructions compose flexibly but provide indirect, coarse-grained control, while post-training adapts behavior more strongly at the cost of repeated updates and potential forgetting.The paper frames this as a tension between composability and learned control without continually changing the backbone.Source: §1 · pp. 1–2

Contributions

  • Composable alignment learns requirements independently as prompts over a frozen backbone, then jointly controls any subset at inference through weighted blending.Unlike task-specific prompt assignment or weight merging, prompt mixtures remain input-side and can be changed per request.Source: §1 · p. 2

Method

  • AlignFormer learns a fixed-length prompt per requirement, and composability regularization transfers textual semantic geometry into prompt space to support meaningful blending.Prompts are stored in a bank; each stage adds a requirement-specific prompt while leaving prior prompts and the backbone untouched.Source: §1 · p. 2

Contributions

  • Ready2Blend is the only frozen-backbone method matching strong post-training methods across two incremental settings, reaching 93.1–98.5% of joint training with competitive backward transfer and 2.6–4.3× less training time.The method supports inference-time weighted personalization and remains stable across different alignment orders.Source: §1 · p. 3

2 RELATED WORK

Prior continual-alignment methods preserve earlier behaviors through sequential backbone updates, while prompt-based methods avoid those updates but generally assign standalone prompts to tasks. Ready2Blend instead grounds requirement prompts in language and makes them jointly composable.

Continual Alignment through Post-training

  • Continual-alignment methods such as CPPO and LifeAlign consolidate requirements into shared model state, so each new requirement updates the backbone and moves the deployed model further from its initial state.CPPO jointly optimizes policy learning and retention; LifeAlign merges stage-wise LoRA updates.Source: §2 · p. 3
  • Weight-delta merging can recombine independently trained objectives, but every mixture rewrites model weights, unlike Ready2Blend’s input-space blending.LifeAlign applies conflict-aware consolidation and is used as the representative LoRA-merging baseline.Source: §2 · p. 3

Prompt-Based Continual Adaptation

  • Prompt-based continual-learning methods steer frozen backbones with few trainable parameters, but assign prompts per task rather than jointly controlling alignment requirements.Ready2Blend grounds prompts in requirement definitions and regularizes their semantic structure for composition.Source: §2 · p. 3

Frozen-backbone Steering Beyond Continual Learning

  • Activation steering and prompt compression encode a single known behavior at inference, without handling requirements that arrive over time or preserving earlier ones as new requirements are added.These methods also lack alignment supervision for learning evolving requirements.Source: §2 · p. 3

3 PROBLEM: LIFELONG ALIGNMENT

Lifelong alignment studies sequentially incorporating new requirements while retaining earlier ones, with supervision revealed stage by stage. The requirements may accompany new tasks or express new preferences for a fixed task.

Problem Setup

  • Lifelong alignment concerns sequentially adapting to evolving requirements while preserving earlier alignment, and the final model is evaluated on each encountered requirement.At each stage, future requirements are unknown and only current supervision is available.Source: §3 · pp. 3–4
  • The two settings differ in whether alignment accumulates across tasks or across preferences within one task, and both involve catastrophic forgetting.The setup represents a model as receiving a natural-language requirement and input, with alignment supervision supplied as preference pairs.Source: §3 · pp. 3–4

Task-Incremental Alignment

  • In task-incremental alignment, each stage adds a new task with its own requirement while retaining behaviors learned for prior tasks.The setting models progressive expansion to new applications or domains.Source: §3 · p. 4

Preference-Incremental Alignment

  • In preference-incremental alignment, the task stays fixed while requirements add preferences that may need to be jointly controlled.The motivating example is adding conciseness to factual consistency in summarization.Source: §3 · p. 4

4 READY2BLEND: ALIGNMENT WITH COMPOSABLE PROMPTS

Ready2Blend represents each alignment requirement as an independently learned prompt over a frozen backbone, then structures prompt space so selected requirements can be composed at inference.

4 READY2BLEND: ALIGNMENT WITH COMPOSABLE PROMPTS

  • Each requirement is learned independently as a prompt, leaving the backbone and previously learned prompts unchanged as requirements are added.This formulation keeps requirements jointly controllable at inference without repeatedly updating the backbone.Source: §4 · p. 4, §4.1 · p. 5

4.1 ALIGNFORMER

  • AlignFormer maps requirements of varying textual lengths to fixed-length continuous prompts that can be arithmetically combined.A frozen sentence encoder represents each requirement; learned query tokens attend to that representation, and the output is projected into the LLM embedding space.Source: §4.1 · p. 5
  • At each stage, shared AlignFormer parameters are trained on the current requirement and its supervision, while the resulting prompt is stored in a bank without later updates.Pairwise preferences are used for exposition, though other alignment supervision is applicable.Source: §4.1 · pp. 4–5
  • At inference, selected bank prompts are blended into one fixed-length prompt and prepended to the textual instruction to steer the frozen LLM.The prompt bank maps each seen requirement to its learned prompt.Source: §4.1 · p. 5
  • Requirements can be activated, removed, or reweighted at inference; prompt length stays fixed, and commutative blending makes composition order-free.Stored prompts remain unchanged, unlike post-training that updates the model.Source: §4.1 · p. 5

4.2 COMPOSABILITY REGULARIZATION

  • Independently learned prompts may occupy incompatible regions of representation space, so arithmetic blending can cancel or distort their encoded requirements.Prompt arithmetic requires a shared geometry for meaningful semantic composition.Source: §4.2 · p. 5
  • Composability regularization transfers textual semantic geometry to prompts through point-wise and pair-wise consistency between prompt representations and requirement definitions.Cached representations of earlier prompts anchor the geometry to prompts actually stored in the bank.Source: §4.2 · pp. 5–6
  • Inference-time blending selects a subset of requirements and combines their prompts with convex weights; non-uniform weights adjust their relative influence for personalization.The weighted average approximately reflects the mixture of meanings in the textual definitions while preserving a single prompt’s length and scale.Source: §4.2 · p. 6

5 EVALUATION

Ready2Blend reaches post-training-level final alignment with a frozen backbone, then supports personalized and order-robust blending. Its composability regularizers structure prompts so independently learned requirements work together.

5 EVALUATION

  • Evaluation covers five datasets across six task-incremental stages and four summarization preferences, comparing continual post-training and prompt-based baselines with Last and BWT metrics.The study also analyzes personalization, alignment-order robustness, and composability regularization.Source: §5 · pp. 6–7

5.1 MAIN RESULTS: TASK- AND PREFERENCE-INCREMENTAL ALIGNMENT

  • Ready2Blend reaches 93.1–98.5% of the MTL upper bound with competitive BWT, zero modified parameters, and four input tokens.Its averaged Last is 0.737 vs. 0.723 on Qwen3.5-9B and 0.692 vs. 0.692 on Llama-3.1-8B; CPPO and LifeAlign take 2.6–4.3× longer to train.Source: Table 1 · p. 7, §5.1 · p. 8
  • In task-incremental alignment, several methods show positive BWT, and Ready2Blend closes the final-performance gap between post-training and existing prompt-based adaptation.The task rubrics share criteria such as helpfulness and clarity, while post-training methods adapt strongly with slight forgetting.Source: §5.1 · p. 8
  • In preference-incremental alignment, Ready2Blend limits forgetting as preferences accumulate over the same task, though blending competing requirements contributes some BWT.The reported blending effect is small, with only abstractiveness declining as more prompts are blended across stages.Source: §5.1 · p. 8

5.2 BEYOND RETENTION: PERSONALIZATION AND ALIGNMENT STABILITY

  • Across 14 simulated user profiles, Ready2Blend achieves higher average personalized performance than Text Prompting by setting blending weights directly to user preferences.The profiles vary weights over abstractiveness, faithfulness, completeness, and conciseness.Source: §5.2 · pp. 8–9, Figure 2 · p. 8
  • Across four preference orderings, Ready2Blend has the lowest variance among learned methods (σ = 0.015), and its Last exceeds every baseline’s best ordering.CPPO varies substantially across orders, while frozen-backbone Text Prompting and DualPrompt are more stable but achieve lower final performance.Source: Figure 3 · p. 8, §5.2 · p. 9

5.3 UNDERSTANDING COMPOSABILITY IN READY2BLEND

  • Composability regularization produces clearer prompt clusters that better match the semantic geometry of textual requirements.Without regularization, prompt representations are poorly aligned and weakly structured.Source: §5.3 · p. 9, Figure 4 · p. 9
  • Point-wise consistency improves Learn, while adding pair-wise consistency improves Last, showing that the two regularizers complement each other.Point-wise consistency positions each prompt according to its own requirement; pair-wise consistency preserves relative geometry so prompts remain compatible when blended.Source: §5.3 · p. 9, Table 2 · p. 9

6 CONCLUSION

Ready2Blend learns each alignment requirement as a prompt over a frozen backbone and blends prompts at inference. Across task- and preference-incremental settings, it matches strong post-training methods in final quality with competitive retention, using four input tokens and less training time.

6 CONCLUSION

  • Ready2Blend learns each requirement as a prompt over a frozen backbone and blends prompts at inference, matching strong post-training methods with competitive retention across both incremental settings.It uses four input tokens and 2.6–4.3× less training time; the conclusion attributes effective blending to transferring requirement geometry through point-wise and pair-wise consistency.Source: §6 · p. 10

A DETAILED EXPERIMENTAL SETUP

  • The appendix provides experimental setup details, including dataset statistics, alignment requirements, baseline configurations, and training and inference settings.Source: A DETAILED EXPERIMENTAL SETUP · p. 14

A.1 DATA STATISTICS

The appendix specifies the continual-alignment data streams, comparison baselines, training setup, and inference-time prompt composition used in the experiments.

A.1 DATA STATISTICS

  • Task-incremental data span six stages (15,619 training; 1,083 evaluation examples), while preference-incremental data span four stages (11,906 training; 1,101 shared evaluation documents).The task stream pairs each stage with a distinct capability and supervision; the preference stream covers abstractiveness, faithfulness, completeness, and conciseness.Source: §A.1 · p. 14

A.2 BASELINE DETAILS

  • The baselines cover joint training, sequential updates, forgetting mitigation, and prompt adaptation; MTL uses DPO with access to all stages throughout training.MTL is the non-incremental reference without the sequential-learning constraint.Source: §A.2 · p. 14, §A.2.1 · p. 15, Table 4 · p. 15

A.2.2 PARAMETER-BASED CONTINUAL LEARNING

  • SeqFT, CPPO, and LifeAlign (Li et al., 2026) update the model sequentially, with LifeAlign merging stage-specific LoRA updates into the backbone.SeqFT has no replay or explicit forgetting mitigation; CPPO optimizes current-stage preference pairs, while LifeAlign uses memory-augmented focalized preference optimization.Source: §A.2.2 · p. 15
  • EWC (Kirkpatrick et al., 2017) penalizes changes to important parameters, whereas GEM (Lopez-Paz & Ranzato, 2017) projects conflicting gradients using memory examples.The LLM adaptations estimate Fisher information from response-token loss for EWC and retain 64 examples per completed stage for GEM.Source: §A.2.2 · pp. 15–16

A.2.3 PROMPT-BASED CONTINUAL LEARNING

  • L2P (Wang et al., 2022b) retrieves prompts from a pool, while DualPrompt (Wang et al., 2022a) combines shared and input-selected prompts; both keep the backbone frozen.L2P activates 48 or 32 soft-prompt tokens by setting; DualPrompt uses eight shared and eight expert tokens.Source: §A.2.3 · p. 16

A.3 TRAINING AND IMPLEMENTATION DETAILS

  • Experiments use Qwen3.5-4B, Qwen3.5-9B, and Llama-3.1-8B-Instruct; AlignFormer and the sentence encoder train or remain frozen as specified, while stored requirement prompts stay fixed.Each requirement uses ten paraphrase variants for composability regularization; training uses two epochs per stage unless otherwise specified, with AdamW and four NVIDIA H200 GPUs.Source: §A.3 · pp. 16–17

A.4 INFERENCE SETTINGS

  • For multiple requirements, prompts are composed before being prepended to the frozen LLM; uniform arithmetic averaging is the default, while weighted averaging supports personalization.Averaging keeps prompt length fixed regardless of how many requirements are selected.Source: §A.4 · p. 17

B EVALUATION DETAILS

Evaluation measures continual alignment across the two settings using common protocols, with retention and final performance reported separately.

B EVALUATION DETAILS

  • Backward Transfer (BWT) measures retention, while Last Performance (Last) measures final alignment quality across requirements after the final stage.Higher BWT indicates better retention; negative BWT values indicate backward degradation.Source: B EVALUATION DETAILS · p. 17

B.1 TASK-INCREMENTAL ALIGNMENT

  • Task-incremental evaluation scores each response from 0 to 10 using the original prompt, response, reference answer, and dataset-specific rubric, then averages scores by dataset.The protocol follows LifeAlign (Li et al., 2026).Source: §B.1 · p. 17

B.2 PREFERENCE-INCREMENTAL ALIGNMENT

  • Preference-incremental evaluation uses FineSurE for faithfulness, completeness, and conciseness, and lexical novelty for abstractiveness, macro-averaging each metric over examples.Lexical novelty is the average proportion of novel 1-, 3-, and 5-grams relative to the source article.Source: §B.2 · p. 17

C.1 ALIGNFORMER TRAINING WITH SFT LOSS

The SFT experiment tests whether AlignFormer can use standard cross-entropy supervision instead of relying exclusively on DPO.

C.1 ALIGNFORMER TRAINING WITH SFT LOSS

  • The preference-incremental experiment replaces the default DPO alignment objective with cross-entropy loss and evaluates Qwen3.5-9B and Llama3.1-8B.The paper presents this as evidence that Ready2Blend can use standard SFT supervision.Source: §C.1 · p. 18
  • 0.653 vs 0.578 Last and 0.000 vs −0.121 BWT on Qwen3.5-9B for AlignFormer (SFT) versus SeqFT.The reported comparison indicates improved final performance and retention without updating backbone parameters.Source: Table 5 · p. 18, §C.1 · p. 18
  • 0.618 vs 0.571 Last and −0.003 vs −0.127 BWT on Llama3.1-8B for AlignFormer (SFT) versus SeqFT.The paper reports that SFT-trained AlignFormer also improves both measures on this backbone.Source: Table 5 · p. 18, §C.1 · p. 18

C.2 ROBUSTNESS TO LLM JUDGE CHOICE

The additional experiments examine judge robustness, prompt length, smaller backbones, personalization, blending operators, and performance as requirements accumulate.

C.2 ROBUSTNESS TO LLM JUDGE CHOICE

  • Ready2Blend achieves the highest or tied-highest task-incremental last performance across the three additional LLM judges.The evaluation keeps generated responses and the protocol unchanged while varying the judge.Source: Table 6 · p. 18, §C.2 · p. 18

C.3 STUDY OF THE IMPACT OF PROMPT LENGTH

  • k = 4 provides the best overall performance across prompt lengths, and longer prompts yield no consistent gains in Last or BWT.The result holds across the two evaluated alignment settings.Source: §C.3 · p. 18, Table 7 · p. 19

C.4 EVALUATION ON A SMALL-SCALE LLM

  • On Qwen3.5-4B, Ready2Blend outperforms DualPrompt and L2P on both BWT and Last averaged across the two alignment settings.The paper reports this as evidence that the prompt-based mechanism remains effective with a smaller backbone.Source: Table 8 · p. 19, §C.4 · p. 19

C.5.3 BEHAVIOR UNDER ACCUMULATED ALIGNMENT REQUIREMENTS

  • In task-incremental alignment, performance on earlier requirements remains largely stable and sometimes improves as more requirements are incorporated.Figure 5 tracks each requirement from introduction through the final stage.Source: Figure 5 · p. 20, §C.5.3 · p. 20

C.5.1 PERSONALIZED PREFERENCE BLENDING

  • Personalization is evaluated with 14 synthetic preference profiles whose normalized weights combine scores for four alignment requirements.Four profiles use one-decimal precision and ten use two-decimal precision; the same profiles are applied across methods.Source: §C.5.1 · p. 20

C.5.2 VECTOR ARITHMETIC FOR BLENDING

  • Arithmetic averaging outperforms concatenation on Last and BWT while keeping prompt length fixed as requirements are composed.The comparison covers Qwen3.5-9B under both task- and preference-incremental alignment.Source: Table 10 · p. 20, §C.5.2 · p. 20

C.5.3 BEHAVIOR UNDER ACCUMULATED ALIGNMENT REQUIREMENTS

  • Most preference objectives remain stable or improve as requirements accumulate, but abstractiveness declines after faithfulness, completeness, and conciseness are added.The reported pattern is consistent with a trade-off between lexical novelty and source-grounded, information-preserving summarization; degradation is not uniform across objectives.Source: §C.5.3 · pp. 20–21

C.6 TRAINING AND INFERENCE EFFICIENCY

Ready2Blend reduces training time and prompt-token overhead while retaining competitive continual-alignment performance. Its prompt length stays fixed when requirements are composed, unlike methods that change the backbone parameters.

C.6 TRAINING AND INFERENCE EFFICIENCY

  • LoRA adapters can be merged into the backbone without adding inference prompt tokens, but the resulting backbone parameters differ from the original frozen model.Training time is reported over the full continual sequence under the same hardware configuration.Source: §C.6 · p. 21
  • 1.94 h versus 8.34 h for CPPO and 8.33 h for LifeAlign; Ready2Blend also has higher last performance in evaluated Qwen3.5-9B settings.Across the two alignment settings, CPPO and LifeAlign require up to 4.3× the training time.Source: §C.6 · p. 21
  • Four additional prompt tokens at inference compare with 16 for DualPrompt and 32–48 for L2P, and remain fixed regardless of the number of composed requirements.Arithmetic averaging preserves prompt length.Source: §C.6 · p. 21

D PROMPTS

The appendix specifies shared inference templates and how textual requirements are supplied, then describes the prompts and criteria used to evaluate alignment in both settings.

D.1 INFERENCE PROMPTS

  • Personalized Text Prompting assigns each preference requirement a user-specific importance score from 0 to 100 and omits zero-weight requirements.The scores correspond to weights used to compute the personalized objective.Source: §D.1 · p. 21
  • Shared fixed templates provide the default inference prompts for task- and preference-incremental alignment, with the input document replacing the {document} placeholder.The preference template specifies a shared summarization task, while the task-incremental template gives a general response-generation instruction.Source: §D.1 · p. 21, Table 12 · p. 22
  • Text Prompting appends all requirements introduced through the current stage to the default prompt; Ready2Blend training also uses paraphrases of those criteria.Table 13 describes the stage-wise requirements appended to the baseline prompt.Source: §D.1 · p. 21, Table 13 · p. 22

D.2 EVALUATION PROMPTS

  • Preference-incremental factual consistency is assessed with FineSurE, which identifies sentence-level factuality errors by comparing a generated summary with its source document.The FineSurE prompt uses article and summary placeholders.Source: §D.2 · p. 23, Table 14 · p. 23
  • Task-incremental evaluation uses a common prompt with placeholders for the original user prompt, generated response, and reference answer.The full prompt is provided in Table 15.Source: §D.2 · p. 23, Table 15 · p. 24
  • Dataset-specific evaluation criteria are used for task-incremental alignment.These criteria are presented in Table 16.Source: §D.2 · p. 23, Table 16 · p. 24
Loading 2609.39365v1…