PaperlayerView exact revision on arXiv

Source-linked AI summary

From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning

Fengyuan Liu, Yue Wang, Hangxi Guo, Fengyuan Liu, Chenxu Wu, Yanguang Liu, Mengnan Du

arXiv:2610.11373v1cs.LGcs.CL

TL;DR

Continual learning asks LLMs to acquire new skills and knowledge while preserving existing capabilities, but sequential prompt optimization can forget earlier knowledge and accumulate task-specific rules. EFRE evolves a repertoire of prompt-based functions, refining compatible updates and creating new functions when updates conflict. It reports improved continual-learning performance over GRPO and gains in a minimal agent system.

  • Problem

    Prompt optimization can adapt LLMs without parameter updates, but its direct use in sequential continual learning is questioned because rewrites can overwrite earlier knowledge and accumulate task-specific rules.

    Source: §1 · p. 1

  • Method

    EFRE evolves a repertoire of prompt-based functions, using historical consistency checks to refine compatible functions or create new ones when updates conflict.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    EFRE reaches a final average score of 58.75 with Qwen3-8B and outperforms GRPO and GEPA on the three-task continual-learning sequence.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments report reduced forgetting and improved overall continual-learning performance, with gains also observed in single-task optimization and a minimal agent system.

    Source: §6 · pp. 9–10

    Prompt optimization’s acceptance rule checks current-task performance but not retention on previous tasks, so repeated rewriting can overwrite instructions supporting earlier capabilities.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Continual learning remains challenging for large language models, which must enable models to acquire new skills and knowledge without degrading existing capabilities. Existing approaches typically address this challenge by carefully designing how model parameters are updated. In contrast, prompt optimization avoids costly parameter updates while achieving competitive or even superior performance to reinforcement learning methods such as GRPO on individual knowledge-intensive and reasoning tasks. This raises a natural question: \textit{Can prompt optimization, as an efficient adaptation approach, be directly applied to continual learning?} Our analysis shows that, under sequential task adaptation, it suffers from catastrophic forgetting, while optimized prompts accumulate rules that overfit to local task distributions. To address these limitations, we propose \emph{Evolving Functional REpertoires} (EFRE), which replaces a single prompt with a repertoire of functions that evolves as new tasks arrive: compatible updates refine existing functions, while conflicting updates trigger the emergence of new ones. On a three-task continual-learning stream, EFRE achieves a final average performance 7.50 percentage points higher than GRPO. Moreover, after adaptation to the Bio task, its performance on FinQA decreases by only 1.56 percentage points, compared with 25.10 percentage points for the base prompt optimization method. We further instantiate EFRE in a minimal agent system and observe consistent improvements across different backbone models. Overall, these results demonstrate EFRE's strong performance in continual learning for large language models and highlight its substantial potential for continual learning in advanced agent systems.

1 INTRODUCTION

Prompt optimization offers an efficient alternative to parameter updates, but sequential prompt rewriting can forget earlier knowledge and accumulate task-specific rules. EFRE evolves a repertoire of functions to address these problems, and the reported evaluations show gains in continual-learning and agent settings.

1 INTRODUCTION

  • Sequential prompt updates can overwrite previously learned knowledge and accumulate case-specific rules instead of generalizable principles.These limitations motivate the question of whether prompt optimization can be applied directly to continual learning.Source: §1 · pp. 1–2
  • EFRE evolves a repertoire of prompt-based functions: compatible updates refine existing functions, while incompatible updates can trigger new functions.Historical consistency checks guide refinement and function emergence, with specialization and reuse arising automatically to reduce interference between new and previously learned capabilities.Source: §1 · p. 2
  • EFRE achieves a final average score of 58.75 with Qwen3-8B and outperforms GRPO and GEPA on the three-task continual-learning sequence.After Bio adaptation, forgetting on previously learned tasks is 1.57 points for EFRE, compared with 8.32 for GRPO and 11.82 for GEPA. In a six-task minimal agent system, EFRE scores 18.0 with Qwen3-8B and 36.3 with GPT-5.6-Luna, outperforming ICL and GEPA.Source: §1 · p. 2
  • The paper evaluates EFRE on a three-task continual-learning sequence, single-task benchmarks, and a six-task minimal agent system.It reports consistent improvements over prompt-optimization and GRPO baselines on single-task benchmarks and gains over ICL and GEPA in the agent setting.Source: §1 · p. 2

2 RELATED WORK

The related work presents continual learning and non-parametric optimization as complementary contexts for EFRE. EFRE maintains and evolves external functional contexts rather than updating model parameters.

Continual Learning

  • Continual learning requires acquiring new capabilities while preserving previously learned knowledge and behaviors.Prior approaches address this stability–plasticity trade-off through replay, regularization, parameter isolation, parameter-efficient tuning, and reinforcement-learning-based optimization.Source: §2 · p. 2

Non-Parametric Optimization

  • Non-parametric optimization improves model behavior without parameter updates, and prompt optimization is a representative approach.These methods optimize prompts, contexts, memories, or other external textual states; EFRE organizes the evolving state into a repertoire of functional contexts for sequential tasks.Source: §2 · p. 2
  • Figure 1 depicts EFRE functions as prompts refined with incoming-data feedback, with historical checks guiding refinement, reuse, or emergence of new functions.Incompatible updates prompt a search for another function that can absorb them; if none can, a new function emerges.Source: Figure 1 · p. 3

3 METHODOLOGY

The methodology studies continual learning with a frozen language model and an evolving external textual state. EFRE uses feedback and historical consistency checks to refine, reuse, or add prompt-based functions as tasks arrive.

3.1 PROBLEM STATEMENT

  • The problem setting keeps pretrained model parameters fixed while updating an external textual state to learn new capabilities and preserve earlier ones.Performance is scored by evaluating the frozen model conditioned on that state; held-out evaluation sets are not used to update it.Source: §3.1 · p. 3
  • Plasticity measures performance gain on the current task, while stability measures average performance change on previously encountered tasks.Negative stability indicates average forgetting, zero indicates no net historical change, and positive stability indicates beneficial backward transfer.Source: §3.1 · pp. 3–4

3.2 PROMPT OPTIMIZATION FOR CONTINUAL LEARNING

  • Prompt optimization rewrites the current prompt using task feedback and accepts a candidate only when it improves training-batch and validation performance.Because this acceptance rule does not explicitly check retention on prior tasks, repeated rewriting can overwrite earlier capabilities and accumulate case-specific rules.Source: §3.2 · p. 4

3.3 EFRE: EVOLVING FUNCTIONAL REPERTOIRES

  • A function is a prompt encoding a learned capability, and the repertoire is a set of functions that can serve multiple tasks without advance task assignment.Each function maintains a history of up to H recent examples it successfully absorbed; both functions and repertoire composition evolve over time.Source: §3.3 · p. 4
  • EFRE refines the current function using incoming-task feedback, checks historical consistency, searches other functions after an incompatibility, and creates a new function if none can absorb the update.A candidate must improve training and validation performance; a historically consistent candidate replaces its function, while reuse changes only the matched function and its history. A new function is added only if it improves training and validation performance relative to the pre-update current prompt.Source: §3.3 · pp. 4–5
  • At inference, a textual selector receives the input and current function prompts, selects a function, and directs the frozen model to respond.The selector’s instructions remain fixed while the available prompts change with the repertoire; the selector requires no task labels.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments evaluate EFRE on a three-task continual-learning stream, single-task benchmarks, and ablations of its history, selection, and repertoire mechanisms. They report strong task retention and performance alongside functional specialization and reuse.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers ToolUse → FinQA → SciKE-Bio, single-task benchmarks, and agent tasks, measuring current-task plasticity and changes on previously encountered tasks.Main and agent experiments use Qwen3-8B and GPT-5.6-Luna backbones; ablations use Qwen3-8B as backbone and GPT-5.6-Luna for refinement.Source: §4.1 · pp. 5–7, §4 · p. 5, Table 1 · p. 6

4.2 MAIN RESULTS

  • After Bio adaptation, EFRE† loses 1.56 percentage points on FinQA, versus 25.10 for GEPA and 9.43 for ACE, while its learned functions show specialization and cross-task reuse.With Qwen3-8B, final ToolUse/FinQA/Bio scores are 66.21/68.26/54.83; f6 serves all three tasks, with utilization rates of 17%, 16%, and 19%.Source: Figure 3 · p. 6, §4.2 · pp. 7–8

4.3 ABLATION STUDY AND MECHANISM ANALYSIS

  • With ϵ = 0.1, Bio-stage plasticity is +19.52 points and stability is -0.31, while Last-20 retention yields the least historical degradation after Bio.The moderate tolerance outperforms strict preservation (ϵ = 0) and no historical constraint (ϵ = 1) on Bio-stage plasticity; more retained history does not necessarily improve learning.Source: §4.3 · p. 8
  • The LLM selector has the highest non-oracle plasticity at every stage and better stability than First and Random; Oracle raises Bio-stage plasticity from +19.52 to +24.35 with comparable stability.Embedding-50 gives slightly better Bio-stage stability (-0.29 versus -0.31 points) but lower plasticity throughout.Source: §4.3 · p. 8
  • Across four single-task benchmarks, EFRE scores 49.04 versus 46.22 for GEPA, 44.87 for ACE, and 42.12 for GRPO; GPT-5.6-Luna refinement raises EFRE’s aggregate to 51.26.Repertoires also develop distinct procedures such as evidence decomposition and equation-based reasoning within a single task, without task transitions.Source: §4.3 · pp. 8–9, Table 3 · p. 8
  • Table 4 compares ICL, GEPA, ACE, and EFRE across six interactive environments using two backbones, reporting means and standard deviations.Bold indicates the best result within each backbone, including ties.Source: Table 4 · p. 9

5 EXTENSION: EFRE FOR AGENT CONTINUAL LEARNING

The paper extends EFRE to a minimal agent that learns through environmental interactions while keeping its LLM frozen. Across six interactive tasks, EFRE has the highest mean score with Qwen3-8B and improves averages with both tested backbones.

A Minimal Continual-Learning Agent

  • The agent extension tests whether EFRE remains effective when training data come from environmental interactions, using a shared agent–environment loop.The scaffold is adapted from Continual Learning Bench.Source: §5 · p. 9
  • A single frozen LLM executes tasks, selects functions, and refines them; interaction feedback guides repertoire updates, while historical consistency governs acceptance, reuse, or emergence.Adaptation changes the external repertoire while selector instructions and model parameters remain fixed.Source: §5 · p. 9

Evaluation Results

  • EFRE averages 18.0 with Qwen3-8B versus 9.5 for GEPA and -3.5 for ICL, and averages 36.3 with GPT-5.6-Luna versus 29.5 for GEPA and 22.2 for ICL.With GPT-5.6-Luna, EFRE exceeds the strongest baseline on five tasks and matches ACE on Codebase Adaptation.Source: §5 · p. 9

6 CONCLUSIONS

EFRE keeps the underlying LLM fixed while evolving a prompt-represented repertoire through historical consistency checks. Experiments show reduced forgetting and improved continual-learning performance, with benefits also reported for single-task optimization and agent settings.

6 CONCLUSIONS

  • EFRE evolves a prompt-represented function repertoire while keeping the underlying LLM fixed, using historical consistency checks to refine and reuse functions or add new ones.The conclusion reports that ablations support the roles of historical consistency and function selection.Source: §6 · p. 9
  • Experiments show that EFRE reduces forgetting and improves overall continual-learning performance.The conclusion also reports that functional specialization and reuse benefit single-task optimization.Source: §6 · p. 9
  • A minimal agent-system extension demonstrates gains in interactive environments and supports adaptation from ongoing interactions with dynamic environments.The conclusion describes evolving functional repertoires as a promising approach to continual learning without parameter updates.Source: §6 · pp. 9–10

A DATASETS, BENCHMARKS, AND MODELS

The evaluation covers a three-task continual-learning sequence, four single-task benchmarks, and sequential interactions in six agent environments. The experiments use Qwen3-8B and GPT-5.6-Luna in specified backbone, refinement, and selection roles, alongside an embedding model for function-selection baselines.

A.1 DATASETS AND BENCHMARKS

  • The continual-learning evaluation follows ToolUse → FinQA → SciKE-Bio, while single-task benchmarks are optimized independently.The dataset protocols and sequence follow Harrington et al. (2026).Source: §A.1 · p. 13
  • ToolUse tests tool selection and argument construction, FinQA tests financial numerical reasoning, and SciKE-Bio tests scientific knowledge acquisition and effects on earlier capabilities.ToolUse derives from ToolAlpaca; FinQA questions combine report text and tables with calculations; SciKE-Bio is a multiple-choice biology subset and the final sequence task.Source: §A.1 · p. 13
  • The four single-task benchmarks are HotpotQA, IFBench, AIME-2025, and LiveBench-Math, covering multi-hop QA, instruction following, and mathematical reasoning.AIME-2025 uses 30 problems from the 2025 examinations; LiveBench-Math is the mathematical subset of a regularly updated benchmark.Source: §A.1 · p. 13
  • Continual Learning Bench evaluates sequential adaptation across six stateful environments through interactions that require information accumulation and adaptation to changes.The six environments are Blind Spectrum Monitoring, Codebase Adaptation, Cohort Studies, Database Exploration, Exploitable Poker, and Sales Prediction.Source: §A.1 · p. 13

A.2 MODELS •

  • Qwen3-8B serves as backbone, refinement model, and function selector in corresponding configurations, with its parameters fixed under EFRE.The model has 8.2 billion parameters and a native 32,768-token context window.Source: §A.2 · p. 14
  • GPT-5.6-Luna serves as the backbone, refinement model, and function selector with reasoning effort set to Low; Embedding-20 and Embedding-50 baselines fine-tune on 20 and 50 historical examples per function, respectively.EFRE† uses GPT-5.6-Luna for refinement, whereas EFRE without † uses Qwen3-8B.Source: §A.2 · p. 14

B IMPLEMENTATION DETAILS

The default EFRE configuration specifies function-history, consistency, and repertoire-size settings, while comparison methods use different optimizer and batch-size settings.

B.1 HYPERPARAMETERS

  • The default EFRE configuration uses micro-batches of 3, retains the 20 most recent successfully absorbed examples per function, sets consistency tolerance to 0.1, and caps the repertoire at 50.Experiments are repeated three times; backbone and refinement models follow the reported experimental configurations.Source: §B.1 · p. 14
  • Parameter-updating comparison methods use AdamW with batch size 16, while GEPA and ACE use batches of 3 examples for textual updates.Table 6 summarizes optimizer and batch-size settings for comparison methods.Source: §B.1 · p. 14
  • Embedding-20 and Embedding-50 ablations limit embedding-model training to 20 and 50 historical examples per function, respectively.Source: §B.1 · p. 14

B.2 DATASET SETUP

The LLM continual-learning setup uses disjoint shared splits and the ordered ToolUse → FinQA → SciKE-Bio stream. Agent continual learning instead follows ordered interactions, with updates restricted to information available from completed interactions.

B.2 DATASET SETUP

  • LLM continual learning follows ToolUse → FinQA → SciKE-Bio, with disjoint training, validation, and evaluation splits shared across methods.Training supplies optimization examples and feedback, validation supports update selection, and evaluation measures performance.Source: §B.2 · p. 14
  • AIME-2025 training and validation use 2022–2024 questions, while evaluation uses the 30 questions from 2025.Source: §B.2 · p. 14
  • Agent continual learning uses ordered interactions rather than a static three-way split, and updates use only information from completed interactions.Function histories come from training data, except Last-20+Dval, which also uses validation data for consistency checks.Source: §B.2 · p. 14
  • Table 6 summarizes optimizer and batch-size settings for the comparison methods.Source: Table 6 · p. 15

B.3 EVALUATION PROTOCOLS

Evaluation uses held-out task sets at every learning stage, with stage-local measures separating current-task improvement from changes to earlier tasks. Final scores use equal-weight averages, while agent results use normalized rewards and experiments report uncertainty across three runs.

B.3 EVALUATION PROTOCOLS

  • Evaluation scores every task before adaptation and after each stage; matrix rows represent learner states and columns track tasks, including the initial state.Evaluation examples are excluded from training, prompt refinement, and update selection, and evaluation leaves the learner unchanged; EFRE scores include function selection.Source: §B.3 · p. 15
  • Plasticity measures the current stage’s score change on its incoming task, with positive, zero, and negative values indicating improvement, no change, and degradation.For FinQA, the comparison is its score after FinQA adaptation against its score after ToolUse adaptation.Source: §B.3 · p. 15
  • Stability averages equal-weight changes on previously learned tasks; gains on one task can offset losses on another, and the measure is undefined at the first stage.For the three-task sequence, stability after FinQA covers ToolUse, while stability after SciKE-Bio averages ToolUse and FinQA changes; it differs from cumulative backward transfer.Source: §B.3 · p. 16
  • Final continual-learning performance is the unweighted mean of all task scores after the last stage, while single-task performance averages four benchmarks equally.The single-task aggregate gives each benchmark equal weight regardless of evaluation-set size.Source: §B.3 · p. 16
  • Experiments are repeated three times, with means and sample standard deviations calculated across runs; plasticity and stability use paired within-run scores for uncertainty.Uncertainty for these changes cannot be recovered from separate checkpoint standard deviations alone because within-run scores may be correlated.Source: §B.3 · p. 16
  • Agent evaluation reports normalized rewards, using the same environment-specific reference constants across methods and equal weighting across six environments.These are not accuracy percentages; normalized gain is not used, as it requires a separate stateless evaluation and positive denominator.Source: §B.3 · pp. 16–17

C.1 FULL RESULTS

EFRE† improves final continual-learning performance and retains earlier task performance across both backbones. Its Bio-stage stability is substantially higher than GRPO’s while plasticity remains close.

C.1 FULL RESULTS

  • 63.10 final average with Qwen3-8B, exceeding GRPO by 7.50 points and GEPA by 22.52 points.EFRE† uses GPT-5.6-Luna for refinement; Table 7 reports stage-wise performance and Table 8 reports plasticity and stability.Source: §C.1 · p. 17
  • +19.52 Bio-stage plasticity for Qwen3-8B EFRE† is close to GRPO’s +20.58, while stability is higher at -0.31 versus -8.32.With Qwen3-8B refinement, EFRE reaches a final average of 58.75 and Bio-stage stability of -1.57.Source: §C.1 · p. 17
  • 68.96 final average with GPT-5.6-Luna as backbone, compared with 55.28 for GEPA and 53.54 for ACE.These results show effective adaptation with stronger retention across both backbones.Source: §C.1 · p. 17

C.2 CROSS-MODEL ANALYSIS OF FUNCTIONAL REPERTOIRES

Functional repertoires transfer between Qwen3-8B and GPT-5.6-Luna without further function refinement, though execution-backbone choice affects performance. The best result depends on pairing the repertoire and backbone.

C.2 CROSS-MODEL ANALYSIS OF FUNCTIONAL REPERTOIRES

  • The source backbone learns the repertoire and the target backbone executes it without further function refinement.Same-model configurations provide reference scores, and base scores correspond to the target model before adaptation.Source: §C.2 · pp. 17–18
  • 65.02 final average when transferring the GPT-5.6-Luna repertoire to Qwen3-8B, up from 63.10 with its original execution, with gains on all three tasks.Table 9 reports cross-model repertoire evaluation; its arrows denote source-to-target backbone transfer.Source: Table 9 · p. 18, §C.2 · p. 18
  • 67.13 with GPT-5.6-Luna executing the Qwen3-8B repertoire, exceeding its original execution by 4.03 points and reverse transfer by 2.11 points.The same repertoire performs better with GPT-5.6-Luna in both comparisons, while GPT-5.6-Luna performs best with its own repertoire at 68.96.Source: §C.2 · p. 18, Table 9 · p. 18

C.3 TRAINING COST

This section examines EFRE’s adaptation costs across backbones and refinement backends, and illustrates a function-based prompt workflow that selects functions for required operations, passes verified results between them, and applies specialized checks to validate answers.

C.3 TRAINING COST

  • With Qwen3-8B, default EFRE has the lowest accounted cost at $13.60, using 6.8 GPU-hours versus GEPA’s 6.0 GPU-hours and costing less than GEPA’s $23.52.Their wall-clock times are 3.9 and 3.7 hours, respectively; both EFRE variants use fewer local GPU-hours than SFT, SDFT, SDPO, GRPO, and In-Place TTT in this setting.Source: Table 10 · p. 19, §C.3 · p. 19
  • With GPT-5.6-Luna as backbone, EFRE† costs $23.04, less than ACE’s $31.68 but more than GEPA’s $18.00.EFRE† uses 19.20 million output tokens and takes 4.7 hours, between GEPA’s 3.8 hours and ACE’s 6.5 hours; it is less costly and faster than ACE, while GEPA is less costly and faster than EFRE†.Source: Table 10 · p. 19, §C.3 · pp. 19–20
  • Accounted cost combines API output charges at $1.20 per million tokens and local GPU cost at $2.00 per GPU-hour, excluding API input charges.GPU-hours aggregate allocated device time, while wall-clock time includes execution, API waiting, and orchestration; hosted computation is not counted as local GPU usage.Source: §C.3 · p. 19

E ILLUSTRATIVE EXAMPLES AND PROMPTS

  • The prompt assigns tool execution, evidence retrieval, numerical reasoning, and constraint checking to specialized functions, with sequential handoffs when a request requires multiple operations.The selector chooses by required operation rather than surface similarity; functions stay within their stated responsibilities and preserve verified intermediate results.Source: E ILLUSTRATIVE EXAMPLES AND PROMPTS · p. 26
  • Numerical reasoning extracts quantities, units, periods, and domain restrictions, writes the required expression before calculating, and verifies the result where practical.It distinguishes absolute differences, relative changes, and percentage-point changes, and preserves exact-versus-approximate distinctions through rounding.Source: E ILLUSTRATIVE EXAMPLES AND PROMPTS · p. 26

Related papers

Loading 2610.11373v1…