PaperlayerView exact revision on arXiv

Source-linked AI summary

CASCADE: Case-Based Continual Adaptation for Large Language Models During Deployment

Siyuan Guo, Yali Du, Hechang Chen, Yi Chang, Jun Wang

arXiv:2605.06702v1cs.AIcs.CLcs.LG

TL;DR

LLMs typically stop learning after deployment, despite operating in dynamic environments where continued adaptation matters. The paper formalises deployment-time learning and introduces CASCADE, which uses episodic case memory and contextual-bandit retrieval without modifying model parameters. Across diverse tasks, the framework supports continuous online improvement and no-regret learning, while longer-horizon memory management and learning from failures remain open limitations.

  • Problem

    LLM training and deployment are separated, leaving deployed agents unable to learn from experience; gradient-based adaptation is costly and infeasible for black-box APIs.

  • Method

    CASCADE equips fixed LLM agents with evolving episodic case memory and uses contextual-bandit retrieval to reuse, revise, and retain deployment experiences.

  • Results

    CASCADE achieves continuous online improvement and no-regret learning across diverse tasks while models remain fixed and accessed as black-box APIs.

  • Takeaways & Limitations

    Deployment can function as a continual source of learning through parameter-free adaptation in prompts, memory, retrieval, and other agentic components.

  • Takeaways & Limitations

    Longer-horizon behaviour, memory growth and forgetting, learning from failed attempts, and the required minimum foundational capability remain open limitations.

Abstract

from arXiv · show

Large language models (LLMs) have become a central foundation of modern artificial intelligence, yet their lifecycle remains constrained by a rigid separation between training and deployment, after which learning effectively ceases. This limitation contrasts with natural intelligence, which continually adapts through interaction with its environment. In this paper, we formalise deployment-time learning (DTL) as the third stage in the LLM lifecycle that enables LLM agents to improve from experience during deployment without modifying model parameters. We present CASCADE (CASe-based Continual Adaptation during DEployment), a general and principled framework that equips LLM agents with an explicit, evolving episodic memory. CASCADE formulates experience reuse as a contextual bandit problem, enabling principled exploration-exploitation trade-offs and establishing no-regret guarantees over long-term interactions. This design allows agents to accumulate, select, and refine task-relevant cases, transforming past experience into actionable knowledge. Across 16 diverse tasks spanning medical diagnosis, legal analysis, code generation, web search, tool use, and embodied interaction, CASCADE improves macro-averaged success rate by 20.9% over zero-shot prompting while consistently outperforming gradient-based and memory-based baselines. By reframing deployment as an adaptive learning process, this work establishes a foundation for continually improving AI systems.

1 Introduction

LLM deployment traditionally ends learning, whereas deployment-time learning (DTL) adds a third lifecycle stage for adapting from interaction without changing model parameters. CASCADE implements this through case reuse and contextual-bandit retrieval, with experiments spanning diverse tasks and model scales.

  • The conventional LLM pipeline separates pretraining and finetuning from deployment, where learning essentially stops.
  • Deployment-time learning treats deployment as a third lifecycle stage in which agents adapt from experience without updating the underlying model parameters.
  • CASCADE retrieves, reuses, revises, and retains cases while using a contextual-bandit formulation to balance exploration and exploitation.
  • CASCADE provides a principled deployment-time learning algorithm with provable no-regret guarantees for LLM agents.
  • Experiments cover medical, legal, operational, coding, embodied-interaction, web-search, and tabular-reasoning tasks across models from 4B to 32B parameters.

2 Case-Based Deployment-Time Learning

Deployment-time learning operates online under unknown future tasks, scalar feedback, and fixed or black-box models. CASCADE externalises experience as episodic case memory and adapts retrieval through contextual bandits, retaining successful interactions.

  • Deployment-time learning presents queries as an online stream, requiring agents to reuse prior experience on unseen queries without future-task access.
  • Because parameter updates are often impractical or impossible for black-box APIs, adaptation shifts from model parameters to surrounding agentic components.
  • Unlike single-query test-time adaptation, deployment-time learning accumulates experience and generalises improvements across tasks.
  • Case-based reasoning externalises experience as an explicit episodic memory by retrieving, reusing, revising, and retaining successful cases.
  • CASCADE retrieves cases with a contextual-bandit policy, conditions the frozen LLM on each case, updates retrieval from rewards, and adds successful interactions to memory.

3 Results

CASCADE improves deployment-time performance across single-turn, multi-turn, and real-world tasks without updating the underlying LLM parameters. Its adaptive case retrieval provides strong performance, resource efficiency, generality across model sizes, and improvements in embodied and tool-using settings.

  • Single-Turn Tasks: CASCADE increases average success rate from 63.76% to 66.68% over NP-CBR across benchmarks.The gain is attributed to learning an adaptive retriever policy from task feedback to balance exploration and exploitation.
  • Single-Turn Tasks: CASCADE achieves the best performance across memory-based methods, outperforming REINFORCE+LoRA on 9 of 12 tasks and matching it on the remainder.Retrieving four cases enables CASCADE to surpass REINFORCE+LoRA on the remaining three tasks.
  • Single-Turn Tasks: CASCADE reaches the highest average success rate with less than 4 GB of GPU memory, while REINFORCE+LoRA requires multiple high-end GPUs.This places CASCADE on the Pareto frontier of success rate and resource efficiency.
  • Single-Turn Tasks: CASCADE is strongest in most model-size settings, but all methods fail on MIMIC-IV-MR with Qwen3-4B, indicating a minimum foundational-capability requirement.CASCADE’s lower-bound performance exceeds zero-shot’s upper bound on 9 of 12 tasks.
  • Multi-Turn Tasks: In embodied tasks, CASCADE raises success rates from 62.01% to 67.43% in ALFWorld and from 59.36% to 66.84% in ScienceWorld over NP-CBR.It also improves across ALFWorld task categories and adapts to unseen task distributions.

4 Discussion

The discussion reframes deployment as an essential learning stage in which LLM agents improve through interaction without modifying foundation-model parameters. CASCADE supports this view by shifting adaptation to agentic components and demonstrating parameter-free improvement, while leaving longer-horizon behavior and learning from failure as open challenges.

  • Deployment-time learning treats deployment as a distinct stage where agents continue improving through environmental interaction and feedback.
  • Parameter-free learning shifts adaptation from the foundation model to prompts, memory, retrieval, and other agentic components.
  • CASCADE demonstrates that these agentic adaptations can drive continuous policy improvement across single-turn tasks, multi-turn environments, and real-world case studies.
  • Longer deployment horizons remain untested, with open concerns about memory growth, case management, forgetting, learning from failure, and dependence on foundational capability.
  • CASCADE models case-based reasoning as a contextual bandit, offering computationally efficient deployment-time learning without updating model parameters.

5 Methods

The paper formulates deployment-time learning as online learning with bandit feedback and implements it through CASCADE's evolving episodic memory and adaptive case retrieval. Its regret analysis decomposes performance loss into coverage and retrieval components, yielding sub-linear regret under stated assumptions.

  • 5.1 Problem Formulation: Deployment-time learning models each interaction as observing a query, generating a solution, and receiving binary success feedback over a horizon of T timesteps.The objective is to maximize cumulative reward and achieve sub-linear regret relative to the optimal solution.
  • 5.2 CASCADE: Case-Based Deployment-Time Learning: CASCADE stores successful query–solution–reward tuples in an episodic case bank and retrieves, reuses, revises, and retains cases during deployment.The underlying LLM remains fixed; only successful interactions expand the case bank.
  • 5.2 CASCADE: Case-Based Deployment-Time Learning: CASCADE separates case retrieval from LLM generation, optimizing a lightweight retriever while treating the frozen LLM's response behavior as stationary.This composite policy is represented as µ(c_t|q_t, M_t)p_LLM(a_t|q_t, c_t).
  • 5.2 CASCADE: Case-Based Deployment-Time Learning: The regret decomposition separates coverage gap from retrieval regret, respectively measuring missing suitable cases and failure to select the best available case.The growing case bank can reduce the coverage gap, while adaptive retrieval addresses retrieval regret.
  • 5.2 CASCADE: Case-Based Deployment-Time Learning: Neural-LinLogUCB models retrieval as a contextual bandit, using query–case contexts, an exploration coefficient, regularization, and binary rewards to update the retriever.A pretrained embedding model first recalls candidates, after which the learned retriever selects a case and updates from feedback.
  • 5.3 No-Regret Learning: Sub-linear regret bounds for coverage gap and retrieval regret establish no-regret deployment-time learning without updating the underlying LLM parameters.The guarantees are stated under standard neural contextual-bandit assumptions and sufficient encoder width.

Contents of Supplementary Notes

The supplementary notes situate CASCADE among experiential learning, case-based reasoning, and contextual-bandit approaches for LLM agents. They emphasize CASCADE's principled retrieval mechanism and broad, general evaluation framing while identifying limitations in closely related online methods.

  • LLM Agents with Experiential Learning: Experiential-learning work improves LLM-agent policies from environment interactions, including agentic reinforcement learning that fine-tunes agents with RL on offline task sets.
  • LLM Agents with Experiential Learning: Context-engineering approaches include prompt-based and memory-based methods, with several methods learning prompts or policies from repeated trials or fixed offline training sets.
  • LLM Agents with Experiential Learning: DC and ACE pursue online context adaptation for deployment-time learning but depend on strong underlying LLMs for stable improvements and lack a principled learning mechanism.
  • Case-Based Reasoning with LLMs: Case-based reasoning retrieves, adapts, and retains relevant past cases, connecting conceptually to in-context learning and retrieval-augmented generation.
  • Case-Based Reasoning with LLMs: CASCADE formalizes case retrieval as a contextual bandit to balance exploration and exploitation, provide no-regret learning, and generalize beyond domain-specific CBR methods.
  • Contextual Bandits for LLMs: Contextual bandits provide a principled framework for online decisions conditioned on observed context and have been applied particularly to information retrieval.

B Benchmark Details

DTLBench evaluates deployment-time learning across 16 diverse tasks, including clinical, legal, financial, AIOps, and other decision-support settings. Tasks use online query sequences and task-specific feedback, datasets, and reward functions.

  • Benchmark Scope: DTLBench contains 16 tasks spanning single-turn, simulated multi-turn, and real-world multi-turn settings.The benchmark includes 12 single-turn tasks, 2 multi-turn simulated tasks, and 2 multi-turn real-world tasks.
  • Medical Tasks: Medical tasks cover diagnosis, medication recommendation, specialty referral, and emergency triage using clinical patient information.The tasks use DDXPlus and MIMIC-IV-derived datasets, with options ranging from 19 specialties to 49 pathologies.
  • Evaluation: Task rewards are generally binary, based on exact correctness or successful recommendations, while medication recommendation uses Jaccard similarity above 0.2.Medication ground truth is revealed after submission to simulate real-world feedback.
  • Legal Tasks: Legal tasks evaluate charge and penalty recommendations for multiple defendants from translated Chinese judgment cases.MUD contains 1740 samples and 22 charges, while CMDL is constructed from 2080 samples.
  • Financial Task: Banking intent routing uses 5000 downsampled Banking77 samples with 77 fine-grained intents.Each agent receives a customer-service query and predicts its financial intent.

B.8 Entity-Aware Financial Sentiment Analysis: SEntFiN

The benchmark extends beyond entity-aware financial sentiment to operational diagnosis, text-to-SQL, embodied interaction, web search, and electronic-health-record reasoning. These tasks vary in data source, interaction structure, and feedback definition.

  • SEntFiN: SEntFiN evaluates extracting all entities and their positive or negative sentiments from 2299 multi-entity financial-news headlines.A reward of 1 requires both complete entity extraction and accurate sentiment prediction.
  • Text-to-SQL: SPIDER and BIRD require executable SQL generation from database schemas, with BIRD additionally supplying domain-specific knowledge.The datasets contain 2147 and 1534 samples respectively, and reward successful execution returning the ground-truth answer.
  • Embodied Interaction: ALFWorld and ScienceWorld evaluate embodied task completion under sparse rewards in household and elementary-science environments.ALFWorld requires completion within 30 steps, whereas ScienceWorld rewards an environment score of 100.
  • Deep Search: 2Wiki evaluates multi-hop web question answering through iterative tool use, with an LLM judge assigning binary correctness feedback.The final dataset contains 2500 sampled tasks after removing relatively simple instances.
  • EHR Reasoning: EHR reasoning tasks require agents to interact with a workspace through Python scripts for up to five steps.The final dataset is sampled from MIMIC-III after filtering unverifiable queries.

C.1 Implementation Details

Experiments serve Qwen3 models with fixed inference settings and use ModernBERT-based retrieval components. CASCADE is compared with zero-shot, in-context, non-adaptive case-based, and gradient-based baselines under task-tuned configurations.

  • Model Configuration: Qwen3-series models use bfloat16 inference with temperature 0.1, top-p 0.8, top-k 20, and presence penalty 1.5.Inference uses vLLM and non-thinking mode for faster responses.
  • Black-box Evaluation: Gemini-2.0-flash is evaluated as a black-box model, while Gemini-2.5 models are excluded because safety policies frequently cause refusals on DTLBench tasks.CASCADE’s black-box evaluation excludes three restricted-license clinical tasks.
  • Retrieval: CASCADE retrieval uses gte-modernbert-base for first-stage embeddings and gte-reranker-modernbert-base for reranking.Both retrieval models are built on ModernBERT with an 8,192-token context length.
  • Baselines: Baselines include Zero-shot, ICRL, ICRLPlus, NP-CBR, and REINFORCE+LoRA.NP-CBR removes reward-based adaptive retrieval, while REINFORCE+LoRA combines on-policy reinforcement learning with LoRA finetuning.
  • Hyper-parameter Tuning: CASCADE and REINFORCE+LoRA receive comparable tuning effort, with learning rate tuned per task and CASCADE additionally tuning an exploration coefficient.The paper reports CASCADE as robust to the exploration coefficient.

C.2 Detailed Overall Results

The detailed results table reports means and standard deviations, with selected task-model combinations omitted because of near-zero performance or restricted dataset licensing.

  • Reporting: Results are summarized using mean values and standard deviations across the evaluated settings.
  • Reporting Boundaries: Qwen3-4B results are omitted for MIMIC-IV-MR because performance is near zero.
  • Reporting Boundaries: Gemini-2.0-flash results are omitted for MIMIC-IV-MR, MIMIC-IV-MSR, and MIMIC-IV-TLP because of restricted dataset licensing.

C.3 Ablation Studies

The ablations show that deployment-time feedback, adaptive retrieval, and exploration–exploitation balancing each contribute to CASCADE’s performance. Reward-function modelling and multiple-case retrieval further affect results across tasks.

  • Zero-shot consistently performs worst across tasks, supporting the value of deployment-time learning from environmental feedback.
  • Contextual-bandit adaptation of the retriever adds 2.92 average absolute performance points over non-adaptive CBR.
  • Removing exploration causes a 0.31 average absolute performance drop, showing that retrieval must balance exploration and exploitation.
  • Neural-LinLogUCB achieves the best average performance of 66.68 across tasks, while its neural reward model improves 1.44 over LinLogUCB.
  • Binary-feedback formulation adds a further 1.19 average improvement over NeuralLinUCB.
  • Retrieving more cases and using larger models generally improves CASCADE, but gains remain constrained by the foundation model’s capabilities.

C.5 Human-in-the-loop Results

Human-in-the-loop variants extend CASCADE with expert revision or targeted discovery when the case bank lacks sufficiently relevant cases. Learned discovery strategies outperform random discovery in the reported comparisons.

  • Human-involved revision: Human experts can revise generated solutions, allowing every new case to be safely retained and closing the coverage gap faster.
  • Human-involved revision: Figure S1 compares CASCADE and NP-CBR success rates with and without human revision across 12 single-turn tasks.
  • E-E-D retrieval: E-E-D retrieval uses exploitation for relevant cases, exploration for uncertain cases, and discovery for novel cases absent from the current case bank.
  • E-E-D retrieval: Discovery is triggered below a dynamic threshold based on the 10th percentile of a first-in-first-out queue of 16 recent metric values.
  • E-E-D retrieval: Exploit and UCB yield average relative gains of 8.83% and 8.73% over standard CASCADE, compared with 5.61% for Random.

D Coverage Gap Analysis

The coverage-gap analysis bounds how quickly CASCADE’s episodic memory covers the query space under smoothness, positive-success-probability, and i.i.d.-density assumptions. The resulting scaling depends on intrinsic dimension.

  • Assumptions: The analysis assumes similar queries have similar expected utilities, successful cases occur with probability at least pmin, and queries are i.i.d. with non-negligible local density.
  • Coverage-gap bound: The coverage-gap bound is obtained by combining a lower bound on case-bank growth with a bound on nearest-neighbour distance.
  • Coverage-gap bound: For intrinsic dimension d0 > 1, the accumulated coverage gap scales as ˜O(T^(1−1/d0)); for 0 < d0 ≤ 1, it scales as ˜O(1).
  • Bandit formulation: The contextual-bandit analysis models each action with feature vectors and binary Bernoulli feedback, targeting maximisation of accumulated reward.
  • Bandit formulation: Neural-LinLogUCB uses a deep network for representation learning and a shallow linear head for uncertainty estimation and exploration.

E.3 Regret Analysis

The regret analysis presents Neural-LinLogUCB as a neural contextual-bandit method with binary feedback and derives guarantees under standard regularity and NTK assumptions. The paper also notes a theoretical cost relative to continuous-feedback methods.

  • Algorithm: Neural-LinLogUCB selects actions by maximising an upper-confidence-bound score, then updates its linear head, neural network, and uncertainty matrix from observed rewards.
  • Assumptions: The analysis assumes normalised feature vectors, stable input gradients, positive sigmoid monotonicity, and a positive-definite neural tangent kernel.
  • Regret guarantee: The regret theorem additionally assumes a bounded optimal linear-head norm and selects exploration coefficients for confidence level δ.
  • Regret guarantee: A sufficiently wide neural network yields a regret bound of ˜O(B T^(?)) under the stated theorem conditions.
  • Comparison: The proposed bound is worse than Neural-LinUCB’s O(B√T) bound because binary feedback differs fundamentally from continuous feedback.
  • Practical implementation: Practical CASCADE reduces computation by using stochastic updates for the linear head and infrequent single-step encoder updates.

E.4 Proof of the Main Results

The proof establishes confidence bounds for Neural-LinLogUCB despite representation-learning bias, then uses them to bound instantaneous reward gaps and cumulative regret. Its argument combines neural-network linearisation, matrix bounds, and contextual-bandit lemmas.

  • Supporting lemmas: The proof first invokes prior neural contextual-bandit lemmas to control neural-network linearisation, learned representations, gradients, and design matrices.These ingredients support subsequent confidence and regret bounds under the algorithm’s stochastic-process and norm assumptions.
  • Representation control: Lemma 1 approximates the reward function near the initial parameter with a linear form, while later bounds control the parameter distance from that initialization.The proof uses this approximation to separate representation-learning effects from the linear contextual-bandit estimation problem.
  • Confidence bound: Representation learning introduces bias into reward estimation, so the analysis uses a bias-corrected confidence bound rather than directly applying the standard LinUCB argument.The proof explicitly identifies the mismatch between learned and true representations as the source of the bias term.
  • Confidence bound: Lemma 9 bounds the distance between the estimated parameter and the optimal parameter with high probability, using spectral comparisons involving the sigmoid’s strong monotonicity constant.The proof sandwiches the relevant matrix between scaled design matrices before applying concentration and norm bounds.
  • Regret analysis: The proof decomposes the instantaneous latent reward gap into separate terms and bounds each using Lipschitz, Cauchy–Schwarz, initialization, representation, and confidence arguments.This decomposition culminates in an upper bound for the instantaneous latent reward gap.
  • Regret analysis: The resulting instantaneous-gap bound is aggregated to obtain a cumulative regret bound for Neural-LinLogUCB.The final step uses the confidence-radius maximum together with earlier lemmas and absolute constants independent of other parameters.
Loading 2605.06702v1…