CASCADE: Case-Based Continual Adaptation for Large Language Models During Deployment

Motivation

• The conventional pretraining–finetuning pipeline largely stops learning after deployment, limiting agents’ adaptability, robustness, and long-term performance.Gradient-based adaptation is costly at LLM scale and often infeasible for black-box API services because it requires backpropagation through model parameters.Source: §1 · pp. 1–2

Deployment-time learning

• Deployment-time learning adds a third lifecycle stage that adapts prompts, memory, tools, and decision mechanisms through online interaction while keeping the foundation model fixed.Agents observe tasks, produce solutions, receive scalar success or failure feedback, and optimize long-term performance over individual-error reduction.Source: Figure 1 · p. 2, §1 · p. 2

CASCADE

• CASCADE retrieves, revises, evaluates, and retains successful past cases as episodic memory, using a contextual-bandit retriever to balance exploitation and exploration.The contextual-bandit formulation provides a principled deployment-time learning algorithm with provable no-regret guarantees.Source: §1 · p. 2, Figure 2 · p. 3

Empirical scope

• Across diverse tasks and 4B–32B model scales, deployment-time learning enables fixed, black-box LLM agents to improve continuously from interaction experience.Tasks include medical diagnosis, legal analysis, operational reasoning, code generation, embodied interaction, web information seeking, and electronic-health-record tabular reasoning.Source: §1 · p. 3

• DTL operates online on unseen queries, using only binary success or failure feedback while keeping the deployed LLM parameters fixed.Adaptation therefore shifts from model parameters to agentic components surrounding the foundation model.Source: §2 · pp. 3–4
• Unlike test-time methods that improve individual queries or systems trained once before deployment, DTL targets long-term policy improvement across task streams.The distinction is that DTL accumulates interaction experience and generalises improvements across future tasks.Source: §2 · p. 4
• Case-based reasoning externalises experience as an explicit episodic memory by retrieving, reusing, revising, and retaining successful cases without changing model parameters.This memory-centric mechanism supports interpretability, flexibility, and computational efficiency under DTL constraints.Source: §2 · pp. 3–4
• CASCADE retrieves a past case with a contextual-bandit policy, conditions the frozen LLM on the query and case, and updates retrieval using observed reward.Successful interactions are retained as new cases, progressively expanding memory coverage and improving selection of useful prior experience.Source: §2 · p. 4

3.1 Results on Single-Turn Tasks

• CASCADE outperforms REINFORCE+LoRA on 12 single-turn tasks while using less than 4 GB GPU memory versus multiple high-end GPUs.It achieves the highest average success rate and lies on the success-rate/resource-efficiency Pareto frontier.Source: Figure 3 · p. 5, §3.1 · p. 6
• CASCADE raises average single-turn success from 63.76% for NP-CBR to 66.68% through adaptive retrieval learned from task feedback.It outperforms REINFORCE+LoRA on 9 of 12 tasks and surpasses it on the remaining three when retrieving four cases.Source: Figure 3 · p. 5, §3.1 · p. 6
• CASCADE generalises across Qwen3 model sizes, but effective deployment-time learning requires sufficient backbone capability because all methods fail on Qwen3-4B medication recommendation.Its lower-bound performance exceeds Zero-shot’s upper bound on 9 of 12 tasks.Source: §3.1 · p. 6, Figure 4 · p. 7

3.2 Results on Multi-Turn Tasks

• In embodied multi-turn tasks, CASCADE improves NP-CBR from 62.01% to 67.43% in ALFWorld and from 59.36% to 66.84% in ScienceWorld.It remains strongest across ALFWorld task categories, adapts to unseen tasks at nearly twice Zero-shot success, and improves ReAct by 0.98% and 3.15%.Source: §3.2 · pp. 8, 10, Figure 5 · p. 9
• For web-based deep search, CASCADE improves NP-CBR from 45.94% to 50.06% with local RAG and from 53.29% to 66.51% with web search.Deployment-time learning increases tool calls per query, retrieving more evidence relevant to target questions.Source: §3.2 · pp. 10, 12, Figure 6 · p. 11
• On MIMIC-III tabular reasoning, CASCADE reaches 55.76% success versus 44.02% for NP-CBR and 20.75% for Zero-shot.It consistently leads across task difficulties while reducing debugging rounds through improved API invocation.Source: Figure 6 · p. 11, §3.2 · p. 12

• CASCADE demonstrates no-regret online improvement by treating deployment as a distinct learning stage without updating model parameters.This reframing addresses the gap between offline training on static corpora and deployment in dynamic, open-ended environments.Source: §4 · p. 12
• Parameter-free learning shifts adaptation from the foundation model to prompts, memory, and retrieval, enabling continuous policy improvement across diverse settings.The design suits deployments lacking gradient access, computational resources, or transparency into black-box APIs.Source: §4 · pp. 12–13
• Future work must address longer-horizon memory growth, case management and forgetting, learning from failed experiences, and the difficulty of adaptation when zero-shot performance is near zero.The current framework primarily learns from successful experiences and depends on a minimum level of underlying-model capability.Source: §4 · p. 13
• CASCADE models case-based reasoning as a contextual bandit, making deployment an efficient and complementary source of continual learning beyond scale, data, and architecture.Source: §4 · p. 13

5.1 Problem Formulation

• Deployment-time learning is modeled over T interactions, where each query receives a natural-language solution and binary reward, with the objective of minimizing pseudo-regret.No-regret learning requires sub-linear regret, so average performance asymptotically approaches that of an oracle selecting the optimal solution.Source: §5.1 · p. 13

5.2 CASCADE: Case-Based Deployment-Time Learning

• CASCADE retrieves, reuses, revises, and retains successful query–solution cases in an episodic memory without modifying the underlying LLM.A retriever selects cases from the bank, the LLM generates a solution conditioned on the query and case, and only successful new cases are retained.Source: §5.2 · pp. 14, 16
• Neural-LinLogUCB trains a lightweight retriever online from binary feedback by combining a Transformer encoder, linear reward head, and uncertainty-based exploration.A fixed embedding model recalls candidate cases, while an online-trained cross-encoder reranker selects among them; the LLM parameters are not back-propagated.Source: §5.2 · pp. 14–16

5.3 No-Regret Learning

• The regret analysis separates CASCADE’s total regret into a case-bank coverage gap and retrieval regret, each receiving a sub-linear bound under corresponding assumptions.Coverage improves as successful experiences cover recurring, similar queries, whereas retrieval regret reflects failure to select the most useful available case.Source: §5.2 · p. 14, §5.3 · pp. 16, 18
• CASCADE achieves no-regret deployment-time learning with sub-linear total regret while the backbone LLM remains frozen.The guarantee combines a coverage gap that decreases as successful experiences accumulate with retrieval regret controlled by Neural-LinLogUCB, under the paper’s stated assumptions.Source: §5.3 · pp. 16, 18

Contents of Supplementary Notes

• The supplementary notes report DTLBench summary statistics, including the maximum interaction steps allowed per task, and provide references.The maximum-step statistic describes the interaction budget permitted by each task environment.Source: Table S1 · p. 25

A Comparison with Existing Works

• Agentic reinforcement-learning methods such as PPO3 and GRPO4 optimise LLM parameters offline and assume repeated rollouts, unlike single-reward online deployment-time learning.These approaches use backpropagation through LLM parameters, making them unsuitable for continual adaptation across task streams.Source: A Comparison with Existing Works · p. 25
• Prompt- and memory-based methods often rely on repeated trials, fixed offline data, short-term memory, or static deployment memories rather than continual gradient-free adaptation.ICRL13 evolves prompts across tasks but is constrained by short-term memory, while Agent KB14, Memento15, and FLEX16 remain static during deployment.Source: A Comparison with Existing Works · p. 25
• DC17 and ACE18 are important online deployment-time-learning precursors, but they depend on strong underlying LLMs and lack a principled learning mechanism.Their robustness may therefore vary across model scales and deployment scenarios.Source: A Comparison with Existing Works · pp. 25–26
• CBR retrieves, adapts, and retains structured cases in evolving episodic memory, unlike static ICL demonstrations or unstructured text retrieved by RAG.CBR generally preserves performance comparable to ICL and typically does not fall below zero-shot LLM performance, even when retrieved cases are less relevant.Source: A Comparison with Existing Works · p. 26
• Prior LLM-CBR systems target data science, software testing, drug–drug interaction prediction, or deep research, including DS-Agent27, Re428, CBR-DDI29, and Memento15,30,31.Memento15,30,31 is reported to achieve state-of-the-art performance on the GAIA leaderboard32.Source: A Comparison with Existing Works · p. 26
• CASCADE formulates case retrieval as a contextual bandit, balancing exploration and exploitation to enable no-regret learning across diverse agentic tasks.This principled mechanism distinguishes CASCADE from prior empirical CBR methods and supports a general, scalable framework.Source: A Comparison with Existing Works · p. 26
• Unlike contextual-bandit methods INSTINCT36, EASE38, PILOT39, and Poon et al.40, CASCADE decouples representation learning from uncertainty estimation for end-to-end reward-learning optimisation in CBR-based deployment-time learning.The compared methods rely on a fixed feature extractor, whereas CASCADE studies deployment-time learning under a CBR framework with LLMs.Source: A Comparison with Existing Works · p. 26

B Benchmark Details

• DTLBench contains 16 tasks—12 single-turn, 2 multi-turn simulated, and 2 multi-turn real-world—with shuffled online query sequences except timestamp-ordered LFD.The benchmark statistics are summarized in Table S1.Source: B Benchmark Details · p. 27

B.1 Medical Diagnosis: DDXPlus

• DDXPlus provides 3136 synthetic medical-diagnosis samples requiring prediction among 49 pathologies from patient profiles.The dataset merges its preprocessed validation and test sets and serves as a proof of concept for deployment-time learning in diagnosis.Source: §B.1 · p. 27

B.2 Medication Recommendation: MIMIC-IV-MR

• MIMIC-IV-MR contains 2881 real-world medication-recommendation samples with 122 available medications, requiring recommendations based on diagnoses and procedures.Evaluation uses Jaccard similarity against the ground-truth prescription, with ground truth revealed after submission.Source: §B.2 · p. 28

B.3 Medical Specialty Referral: MIMIC-IV-MSR

• MIMIC-IV-MSR comprises 2115 real-world referral samples requiring selection of the most appropriate specialty from 19 options based on patient profiles.Exact matching against the ground truth determines the reward.Source: §B.3 · p. 28

B.4 Triage Level Prediction: MIMIC-IV-TLP

• MIMIC-IV-TLP contains 2200 triage samples requiring prediction of Emergency Severity Index levels from patient profiles.A prediction receives reward 1 only when it exactly matches the ground truth.Source: §B.4 · pp. 28–29

B.5 Multi-Defendant Legal Charge Prediction: MUD

• MUD contains 1740 translated legal cases with 22 charges, requiring charge recommendations for every defendant in each fact description.The dataset uses the original training split and is translated into English by prompting gpt-4o.Source: §B.5 · p. 29

B.6 Penalty Legal Prediction: CMDL

• CMDL provides 2080 translated multi-defendant legal cases requiring recommendations from six penalty types, potentially with multiple penalties per defendant.The final dataset is constructed from the CMDL-big test split after cleaning and filtering.Source: §B.6 · p. 29

B.7 Financial Costumer Intent Routing: Banking77

• Banking77 uses 5000 randomly downsampled customer-service queries spanning 77 fine-grained intents for financial intent routing.Downsampling reduces the computational cost of evaluating the original dataset, and each query is routed to its corresponding intent category.Source: §B.7 · p. 30

B.8 Entity-Aware Financial Sentiment Analysis: SEntFiN

• SEntFiN contains 2,299 multi-entity financial-news instances requiring all entities and their positive or negative sentiments to be identified, with reward 1 only for completely correct outputs.The dataset is constructed from the original training split and targets entity-level financial sentiment understanding.Source: §B.8 · pp. 30–31

B.9 AIOps Root Cause Analysis: RCA; B.10 AIOps Log Fault Diagnosis: LFD

• RCA contains 2,925 Huawei-forum log samples requiring alert-type prediction, while LFD uses 3,000 chronologically selected Alibaba Cloud log samples for CPU, memory, or hardware fault diagnosis.Both AIOps tasks formulate log interpretation as fault classification, with correct predictions receiving reward 1.Source: §B.9 · p. 31, §B.10 · p. 31

B.11 Text-to-SQL: SPIDER; B.12 Knowledge-Augmented Text-to-SQL: BIRD

• SPIDER and BIRD provide 2,147 and 1,534 text-to-SQL samples, respectively, with BIRD additionally supplying domain knowledge; both use execution accuracy with reward 1 for returning the ground-truth answer.SPIDER uses its test split, whereas BIRD uses its development split.Source: §B.11 · pp. 31–32, §B.12 · p. 32

B.13 Household Embodied Decision Making: ALFWorld

• ALFWorld combines 1,866 sampled training tasks with 134 unseen tasks and limits interaction to 30 steps, rewarding successful completion within that horizon.The benchmark covers six household task types and uses the original ALFWorld reward function.Source: §B.13 · p. 32

B.14 Scientific Embodied Decision Making: ScienceWorld

• ScienceWorld contains 1,857 tasks across 11 scientific task types, with recommended 10–30-step horizons and reward 1 when the environment score reaches 100.Tasks are formed by randomly sampling approximately 200 variations per selected type.Source: §B.14 · p. 33

B.15 Web-based Deep Search: 2Wiki

• 2Wiki constructs 2,500 multi-hop web-search tasks after removing simpler questions, limits tool interactions to five steps, and uses an LLM-as-judge reward based on matching the ground truth.The judge is gpt-4o-mini, following previous works15,57.Source: §B.15 · p. 33

B.16 Complex Tabular Reasoning on Electronic Health Records: MIMIC-III

• MIMIC-III uses 2,500 filtered EHRSQL tasks requiring Python-based workspace interaction within five steps, with execution accuracy determining binary reward.The dataset follows EHRAgent60 to remove unverifiable queries.Source: §B.16 · p. 34

C Supplementary Results

• The supplementary-results section describes the experimental setup and reports additional empirical results.It follows the benchmark descriptions above.Source: C Supplementary Results · p. 34

Model Configuration

• Qwen3-series models61 use vLLM with bfloat16, temperature 0.1, top-p 0.8, top-k 20, presence penalty 1.5, and non-thinking inference.Qwen3-30B-A3B-Instruct-2507 is reserved for deep-search tasks because of its tool-calling capabilities, while the original Qwen3-4B is replaced by its instruct variant.Source: §C.1 · p. 34
• Gemini-2.0-Flash uses API inference with temperature 0.05 and top-p 0.8, while Gemini-2.5-series models63 are excluded because safety refusals frequently occur on DTLBench tasks.Source: §C.1 · p. 34

Retrieval Models

• Retrieval uses gte-modernbert-base for first-stage embeddings and gte-reranker-modernbert-base for reranking, both built on ModernBERT with 8,192-token context64,65,66.Source: §C.1 · p. 35

Baselines

• Baselines include zero-shot, ICRL13, ICRLPlus13, NP-CBR27,15, and REINFORCE+LoRA67,68, while DC17 and ACE18 are omitted because they require stronger backbones and may degrade weaker models.ICRL uses recent trials, ICRLPlus retains recent correct trials, and NP-CBR isolates the effect of CASCADE’s bandit-based retrieval policy.Source: §C.1 · pp. 35–36

Implementation details of CASCADE

• CASCADE tunes each task’s learning rate with exploration fixed at zero, then tunes exploration; learning rate is most critical, while exploration remains robust.The authors recommend small-scale online hyper-parameter experiments before deployment, with tuning effort comparable to REINFORCE+LoRA.Source: §C.1 · p. 36

C.2 Detailed Overall Results

• Table S4 summarizes all main results with mean values and standard deviations.Source: §C.2 · p. 36
• Qwen3-4B omits MIMIC-IV-MR because of near-zero performance, while gemini-2.0-flash omits three MIMIC-IV datasets because of restricted licensing.Source: §C.2 · p. 36

C.3 Ablation Studies

• The ablation benchmark evaluates CASCADE variants on 12 single-turn DTLBench tasks using five random seeds, with all Table S5 results based on Qwen3-32B.The variants include Zero-shot, removal of Neural-LinLogUCB, no exploration, and replacements using LinLogUCB, NeuralLogUCB, or Neural-LinUCB.Source: §C.3 · p. 36, Table S5 · p. 38
• 2.92 average absolute improvement comes from adapting the retriever policy with contextual-bandit reward feedback, while omitting exploration causes a 0.31 average absolute drop.Zero-shot is consistently the poorest variant across tasks, and over-reliance on the learned reward model without exploring uncertain cases reduces performance.Source: §C.3 · p. 38, Table S5 · p. 38
• 66.68 is Neural-LinLogUCB’s best average performance across tasks, exceeding NeuralLinUCB by 1.19 and LinLogUCB by 1.44 on the ablation benchmark.Neural-LinLogUCB decouples representation learning from uncertainty estimation and uses binary feedback, whereas LinLogUCB relies on a linear assumption and NeuralLogUCB remains dependent on pretrained representations.Source: §C.3 · p. 38, Table S5 · p. 38

C.4 Extension of CASCADE to Multiple Cases

• CASCADE’s multiple-case extension selects the top-k cases by UCB scores and consistently outperforms NP-CBR across most tested retrieval settings.The extension applies a combinatorial neural bandit formulation to generalize Neural-LinLogUCB from one retrieved case to multiple cases.Source: §C.4 · p. 39
• CASCADE’s performance steadily improves as the number of retrieved cases increases, closing REINFORCE+LoRA’s single-case advantage on three of twelve tasks by four retrieved cases.The affected tasks are MIMIC-IV-MSR, MIMIC-IV-TLP, and LFD, illustrating that appropriate context engineering can let memory-based learning surpass gradient-based learning.Source: §C.4 · p. 39
• CASCADE generally benefits from more retrieved cases and larger models, but scaling retrieval alone cannot guarantee gains because improvement is bounded by the LLMs’ foundational capabilities.This scaling analysis is reported on a representative subset of single-turn tasks.Source: §C.4 · p. 39
• CASCADE outperforms the best non-parametric baseline NP-CBR in almost all model-size and retrieval-count settings.The comparison is reported as CASCADE’s absolute improvement over NP-CBR in Fig. E1c.Source: §C.4 · p. 39

CASCADE under E-E-D retrieval framework

• 8.83% and 8.73% average relative performance gains over standard CASCADE are achieved by Exploit and UCB, respectively, versus 5.61% for Random.Exploit and UCB also consistently outperform other discovery metrics in counterfactual discovery advantage.Source: Figure S1 · p. 40, §C.5 · p. 41

Human-involved revision

• CASCADE consistently outperforms NP-CBR across all human-revision settings, although both methods can deteriorate on four tasks as case banks grow and diversify.Human revision safely retains a ground-truth-labelled case at every timestep, setting pmin = 1 and accelerating coverage-gap closure.Source: §C.5 · p. 39, Figure S1 · p. 40

CASCADE under E-E-D retrieval framework

• With a fixed 10% discovery budget, CASCADE’s contextual-bandit retriever supports exploitation, exploration, and discovery when sufficiently relevant cases are absent.Discovery directly accesses ground-truth labels, enabling selective human expert assistance within a limited budget; this follows the E-E-D framework in infinitely many-armed bandits [75, 76].Source: §C.5 · p. 41
• CASCADE triggers discovery using Explore, Exploit, or UCB metrics against a dynamic 10th-percentile threshold maintained over a first-in-first-out queue of length 16.Random is the baseline, initiating discovery with probability 10%.Source: §C.5 · p. 41

D Coverage Gap Analysis

• The coverage analysis assumes similar queries have similar expected utilities, every query-case conditioning retains positive success probability, and i.i.d. queries satisfy a mild density condition.These assumptions ensure perturbation stability, nonzero case-bank growth, and recurring nearby queries needed for sub-linear coverage analysis.Source: D Coverage Gap Analysis · pp. 41–42
• Theorem 4 bounds CASCADE’s accumulated coverage gap with high probability under the three stated assumptions, with scaling determined by the intrinsic dimension d0.The analysis relates coverage to nearest-neighbour distances in the case bank and the number of accumulated successful cases.Source: D Coverage Gap Analysis · pp. 42–43
• For d0 > 1, the coverage gap has a ˜O(T^d0) bound, whereas for 0 < d0 ≤1 it has a ˜O(1) bound.Thus, the intrinsic dimension controls whether the bound grows with the deployment horizon or remains bounded.Source: D Coverage Gap Analysis · p. 43

E Neural-Linear Logistic Bandit: Neural-LinLogUCB

• The Neural-LinLogUCB analysis is formulated in the standard neural contextual bandit setting with binary feedback and aims to minimize pseudo-regret.The reward is modeled as Bernoulli with probability given by a sigmoid-transformed unknown latent reward function.Source: E Neural-Linear Logistic Bandit:… · pp. 43–44, §E.1 · p. 44

E.1 Preliminary

• The preliminary model uses a fully connected ReLU network to represent complex reward functions and its gradient features for neural contextual-bandit analysis.The network has depth L ≥2, and the parameterization defines a gradient g(x; ω) used by the algorithm.Source: §E.1 · p. 44

E.2 Neural-Linear Logistic Bandit Algorithm

• Neural-LinLogUCB decouples representation learning from uncertainty estimation by pairing a deep network f(·; ω) with a shallow linear head θ.This design preserves deep networks’ approximation capability while enabling more efficient exploration through the low-dimensional head.Source: §E.2 · pp. 44–45
• At each timestep, the algorithm selects the action with the largest upper confidence bound, then estimates the logistic linear head with L2 regularization and trains the neural network by cross-entropy loss.The confidence matrix uses regularization strength λ, and updates rely on the observed history of feature vectors, actions, and binary rewards.Source: §E.2 · p. 45

Algorithm

• The theoretical algorithm selects arms using an upper-confidence rule, updates its logistic linear head, trains the neural network, and updates the design matrix each round.Its analysis follows neural contextual-bandit work based on the neural tangent kernel.Source: §E.3 · pp. 45–46

Assumptions

• The guarantee assumes normalized, structured features, Lipschitz gradient stability, positive strong monotonicity, and a positive-definite NTK matrix.The NTK condition requires H ⪰ λ0I for some λ0 > 0.Source: §E.3 · pp. 46–47

Regret guarantee

• Under Assumptions 4–7 and a bounded optimal head, Neural-LinLogUCB achieves its stated high-probability regret bound when the network width is sufficiently large.The guarantee holds with probability at least 1 − δ and requires m ≥ poly(L, d, 1/κσ, Lσ, log(1/δ), log(TK)).Source: §E.3 · p. 47

Comparison with Other Neural Contextual Bandits

• Compared with Neural-LinUCB73, Neural-LinLogUCB has a worse regret bound with additional dependence on κσ because binary feedback differs fundamentally from continuous feedback.The comparison attributes the gap to the binary-feedback setting, analogously to the difference between NCBF-UCB72 and Neural-UCB37.Source: §E.3 · p. 47

Practical Implementation of Neural-LinLogUCB in CASCADE

• For CASCADE, the method replaces MLPs over frozen embeddings with a pretrained Transformer encoder and initializes its linear head from the pretrained reranker.It uses stochastic head updates every step and one mini-batch encoder update every H timesteps to reduce computational cost while retaining adaptation.Source: §E.3 · pp. 47–48

• Neural-network approximation, parameter-distance, gradient, output-linearisation, and representation-vector lemmas control the learned representation’s deviation from its initial linearisation.These lemmas are established using prior neural contextual-bandit techniques and hold with high probability where stated.Source: §E.4 · pp. 48–49
• Matrix concentration and elliptical-potential lemmas provide the auxiliary bounds needed for the confidence analysis and final regret calculation.The matrix results assume bounded feature norms, regularisation, and, for the stochastic-process result, conditionally sub-Gaussian noise.Source: §E.4 · p. 49
• The proof introduces a bias-corrected confidence bound because representation learning creates bias that direct LinUCB-style analysis would otherwise leave uncontrolled.The bias is defined from the representation mismatch and incorporated into the confidence analysis through Lemma 9.Source: §E.4 · p. 50
• Theorem 5 bounds Neural-LinLogUCB’s cumulative regret after bounding its instantaneous latent reward gap.The final regret inequality follows by summing the per-round bound and applying the confidence-radius and matrix lemmas.Source: §E.4 · pp. 54–55