PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

Parth Asawa, Christopher M. Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, Asim Biswal, Vincent Sunn Chen, Frederic Sala, Matei Zaharia, Joseph E. Gonzalez

arXiv:2606.05661v1cs.AIcs.CL

TL;DR

Existing evaluations do not directly test whether systems improve online by discovering environment-specific structure across related tasks. CL-BENCH introduces expert-validated tasks across six domains and a gain metric to measure learning beyond stateless capability. Frontier systems still leave substantial headroom, with naive ICL outperforming dedicated memory architectures on most tasks.

  • Problem

    Existing evaluation proxies do not directly test whether systems improve online by learning environment-specific latent structure across related tasks.

    Source: §1 · p. 2

  • Method

    CL-BENCH evaluates systems on expert-validated tasks with hidden, task-specific structure across six domains and uses gain to isolate online experience from stateless performance.

    Source: §1 · p. 2, §4.1 · p. 6, §4.2 · p. 6

  • Results

    Naive ICL outperforms dedicated memory architectures on most tasks, and even the best system achieves only 25.4% normalized gain over its stateless baseline.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Continual learning in LLM-based agents remains an open problem, and CL-BENCH provides a foundation for tracking progress.

    Source: §7 · p. 10

    The benchmark covers six domains and sequences of tens of instances, and its initial evaluation focuses on context-based memory rather than parametric approaches such as test-time training.

    Source: §6 · p. 9

Abstract

from arXiv · show

Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting), each validated by domain experts and designed so that tasks share a learnable latent structure (codebase layout, disease outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. We evaluate frontier models across several agent architectures, from naive in-context learning (ICL) to dedicated memory systems, introducing a gain metric to isolate learning from prior capabilities. We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this -- in fact, naive ICL outperforms systems dedicated to memory management. CL-Bench is the first benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks and isolate online learning from underlying model capability, showing a need for better continual learning systems.

1 Introduction

Existing evaluations do not directly test whether systems learn environment-specific structure online and use it to improve later performance. CL-BENCH introduces expert-validated tasks and a gain metric for measuring this learning, then finds that frontier systems still struggle to benefit reliably from experience.

1 Introduction

  • Existing memory, long-context, and adaptation evaluations use proxies such as recall, context preservation, or new-task accuracy rather than testing online improvement through shared environmental structure.These measures do not directly establish whether systems learn environment-specific latent structure across related tasks.Source: §1 · p. 2
  • CL-BENCH spans six real-world domains with hidden, task-specific latent structure that systems must discover online, sometimes adapting when that structure drifts.The database example illustrates schema discovery followed by a migration that makes some past knowledge stale.Source: §1 · p. 2, Figure 1 · p. 2
  • The gain metric compares reward with online experience against the same system’s stateless performance to isolate the contribution of accumulated state.This helps distinguish learning from a system’s baseline ability on intrinsically easy or difficult instances.Source: §1 · p. 2
  • Naive in-context learning outperforms dedicated memory architectures on most tasks, and even the best system achieves only 25.4% normalized gain over its stateless baseline.Agents often overfit to recent observations or fail to reuse knowledge across instances.Source: §1 · p. 3

2 Related Work

Prior continual-learning and agent benchmarks evaluate retention, predefined skill reuse, recall, or isolated capabilities, but do not require discovering hidden, recurring structure from experience. CL-BENCH instead builds tasks where that structure is task-specific and online experience is needed to improve later performance.

2 Related Work

  • Continual-learning benchmarks measure retention, forgetting, transfer, predefined skill reuse, or conversational recall, but do not require discovering hidden structure that improves later instances.CL-BENCH’s six expert-validated domains use hidden, task-specific structure not recoverable from pretraining.Source: §2 · p. 3
  • Many agent benchmarks test isolated tasks or iterative performance without shared latent structure, allowing stronger static capability to stand in for continual learning.CL-BENCH designs recurring structure that systems must exploit through online experience.Source: §2 · p. 3

3 Continual Learning Bench

CL-BENCH defines tasks as sequential environments where experience and feedback can support improvement, then applies admission criteria and expert review to ensure that improvement reflects online learning. Its six-domain task set varies in workflow and reward while retaining structured, reproducible evaluation.

3 Continual Learning Bench

  • A CL-BENCH task is a sequence of instances in a shared environment, with systems accumulating experience and feedback as they proceed.A system may combine any model and learning mechanism; tasks may include variants that introduce concept drift.Source: §3 · p. 3

3.1 Task Admission Criteria

  • Tasks require headroom, shared latent structure, and informative feedback so improvement reflects online learning rather than static model capability.The structure must be task-specific, recurring across instances, inferable from experience, and actionable through environmental feedback.Source: §3.1 · p. 4
  • Database Exploration asks agents to answer 40 questions about an obfuscated SQLite database, then adapt when a halfway migration makes prior schema knowledge stale.Schema and data conventions can be learned early and reused, while migration tests whether agents detect and discard stale experience.Source: §3.1 · p. 4

3.2 Task Validation

  • Each task receives author review and independent review by two to three domain experts for realism, reusable knowledge, and measurable learning improvement.Tasks are accepted only after confirmation across all three axes.Source: Figure 2 · p. 4, §3.2 · pp. 4–5
  • The tasks are learnable but remain unsolved by current systems, with dedicated memory systems performing worse than the simplest in-context learning baseline.Even ICL shows deficits, including over-reliance on recent instances and under-weighting relevant earlier variants.Source: §3.2 · p. 5

3.3 Benchmark Construction

  • The benchmark spans six domains, varied reward types, and different interaction styles, with structured outputs for reproducible scoring.The workflows include persistent terminal environments and a structured tool API.Source: §3.3 · p. 5, Figure 3 · p. 5

4 Metric Design

Metric design pairs task-linked reward with gain, comparing stateful performance against a stateless counterfactual to distinguish learning from absolute performance.

4.1 Reward

  • Reward aggregates task-linked instance outcomes; stateless policies use only the current instance, while stateful policies condition on the full history.Because instance difficulty and system baselines vary, high absolute reward alone does not establish learning.Source: §4.1 · pp. 5–6

4.2 Gain

  • Gain subtracts the same system’s stateless reward on each instance from its stateful reward, isolating accumulated-state contribution despite varying difficulty and baselines.The figure presents the stateful-versus-independent performance comparison.Source: §4.2 · p. 6, Figure 4 · p. 6, §4.1 · p. 6

4.3 Cross-Task Normalization

  • Normalized gain divides raw gain by system-specific learning headroom; normalized reward uses a fixed GPT-5.4 ICL reference, making reward scores submission-independent.Learning headroom is the maximum gain available given the system’s stateless baseline.Source: §4.3 · p. 6

5 Evaluating Frontier Systems on CL-BENCH

Across the evaluated systems, full-context ICL with Claude Sonnet 4.6 leads in aggregate normalized reward and gain, while learning dynamics vary by task and memory architecture. ICL variants generally occupy the gain–cost frontier, and systems differ in their ability to reuse prior information versus adapt to new variants.

5 Evaluating Frontier Systems on CL-BENCH

  • The evaluation averages systems over five rollouts per task and compares frontier language models using multiple continual-learning architectures.Architectures include full-context ICL, persistent notepads, retrieval memory, playbooks, and agentic harnesses with automatic context compaction.Source: §5 · p. 7

5.1 Results

  • Learning varies by task: Sales Prediction and Blind Spectrum Monitoring show the clearest gains, while Cohort Studies has the least and several tasks show minimal gain.In Database Exploration, stateless reward often collapses to zero while stateful reward maintains a floor; Figure 5 compares stateful and stateless curves.Source: §5.1 · pp. 7–8, Figure 5 · p. 8
  • Full-context ICL with Claude Sonnet 4.6 achieves the highest aggregate normalized reward (22.3%) and gain (25.4%) of any evaluated system.ICL-based systems occupy three of the top five gain positions; Table 1 reports the aggregate comparison and rollout costs.Source: Table 1 · p. 7, §5.1 · p. 7
  • ICL variants largely occupy the gain–cost frontier, while Claude Code is an exception with the second-highest gain (23.9%) at $38.6 per run.Gemini Flash with ICL offers the best cost efficiency at $7.6 per run with 16.4% gain; ACE costs $62.8 without proportionally higher gain.Source: §5.1 · pp. 8–9, Figure 6 · p. 8
  • Claude Sonnet 4.6 with ICL Notepad shows the most stable learning, whereas ICL and Claude Code show the most plastic learning across variants.Stability measures reuse across variant switches, while plasticity measures adaptation within variants; Figure 7 decomposes normalized gain into these terms.Source: §5.1 · p. 9, Figure 7 · p. 9

6 Limitations

CL-BENCH covers six diverse, expert-validated domains, but does not span all real-world continual-learning scenarios. Its evaluation is bounded by shorter task sequences, a focus on context-based memory, and tasks that may not surface smaller-model failure modes.

6 Limitations

  • The six diverse, expert-validated domains do not fully span real-world continual-learning scenarios.Task sequences contain tens of instances, shorter than the longest deployment horizons practitioners may care about.Source: §6 · p. 9
  • The initial evaluation covers context-based memory paradigms but not parametric approaches such as test-time training.The authors seek to add parametric methods through community contributions as realistic continual-learning methods develop.Source: §6 · p. 9
  • Because tasks require frontier-level capability to perform non-trivially, failure modes of smaller models may not be surfaced.Source: §6 · p. 9

7 Conclusion

CL-BENCH is an expert-validated benchmark spanning six real-world domains, designed to measure improvement through sequential experience while isolating learning from underlying model capability. Evaluation finds substantial headroom: naive in-context learning outperforms dedicated memory architectures on most tasks, and accumulated state can hurt rather than help.

7 Conclusion

  • CL-BENCH spans six real-world domains and uses verifiable rewards and analysis metrics to isolate learning from underlying model capability.The benchmark is intended to support rigorous comparisons of continual-learning strategies and grow through community contributions.Source: §7 · pp. 9–10
  • Naive ICL outperforms dedicated memory architectures on most tasks, while even the best system achieves only 25.4% normalized gain over its stateless baseline.Accumulated state frequently hurts through spurious generalizations and stale beliefs, and more expensive systems fail to translate cost into performance.Source: §7 · p. 10

A.1 Blind Spectrum Monitoring

  • The Blind Spectrum Monitoring task tests whether agents maintain a map of persistent transmitters across scans, including when channels go dormant.The union of channel definitions remains fixed across three stages, while the latent set becomes more observable; reward is mean IoU across 90 scans.Source: §A.1 · pp. 13–14

A.2 Codebase Adaptation

  • Codebase Adaptation tests whether agents reuse repository-specific knowledge about code locations, tests, and conventions across sequential bug-fix issues.The sequence groups nine tablib issues before ten tenacity issues, creating two within-repository learning blocks separated by a repository switch.Source: §A.2 · p. 14

A.3 Cohort Studies

  • Cohort Studies tests whether agents integrate shared survival predictors across studies with different variables, coding conventions, and biased samples.The schedule comprises five study blocks, and agents receive no cross-study dictionary; they must map local columns to prior knowledge.Source: §A.3 · pp. 15–16

A.4 Database Exploration

These task details describe how database exploration, poker, and sales forecasting offer sequential opportunities to learn and reuse task-specific structure. The appendix also reports evaluation procedures and limits on model coverage.

A.4 Database Exploration

  • A.4 Database Exploration tests reuse of discovered schema and data conventions across 40 questions, with a migration after question 20 requiring agents to detect changed information.The database combines obfuscated columns, category-specific units and table structures, and migration changes; reward penalizes exploratory-query use and incorrect answers.Source: §A.4 · pp. 17–18

A.5 Exploitable Poker

  • A.5 Exploitable Poker tests whether agents infer deterministic opponent policies and carry strategy updates across hands and opponent transitions.Stacks reset each hand, and the 120-hand schedule returns to earlier opponent types to test retention and faster readaptation.Source: §A.5 · pp. 18–19

A.6 Sales Prediction

  • A.6 Sales Prediction tests whether agents learn persistent cross-location demand structure to forecast 75 product-location-year combinations, including unseen stores and products.The task describes stable cluster growth rates and price-demand relationships, while rotating visible stores and changing schemas can make the data appear nonstationary.Source: §A.6 · pp. 19–20
  • By instance 4, a learning agent has accumulated three years of data across two stores, fitted a Poisson GLM, and can produce calibrated out-of-assortment forecasts.The passage reports that a stateless agent refitting from a single store-year produces poorly constrained growth slopes and scores significantly lower.Source: §A.6 · p. 20

B.1 Noise Reduction

  • Stateful rollouts are resampled by permuting instance order, and final metrics are averaged across five rollouts with 95% confidence intervals.The stateless baseline is evaluated once per task-system pair because its instances are order-independent; Codex is excepted from five-rollout evaluation due to resource limitations.Source: §B.1 · p. 21

B.2 System Selection Rationale

  • The evaluation prioritizes coverage of distinct memory paradigms over exhaustive model coverage, pairing structured memory systems primarily with GPT-5.4.Smaller models and open-weight systems are outside the paper's scope because exhaustive model-by-memory evaluation would require thousands of additional dollars and runs.Source: §B.2 · p. 21

B.3 Normalized Results Per Task

Figure 8 provides a compact view of normalized reward by system across the six CL-BENCH tasks.

B.3 Normalized Results Per Task

  • Figure 8 presents per-task normalized reward as percentages, with warmer cells indicating stronger relative performance and cooler cells indicating below-baseline performance.The figure uses the same normalization as Table 1 and shows which systems are strongest on which tasks.Source: Figure 8 · p. 21, §B.3 · p. 21

B.4 Task Specific Results

The task-specific results report cumulative reward and gain on each task's native scale, alongside cost and task-level visual comparisons. The supplied captions identify Pareto-frontier plots and stateful-versus-stateless learning curves, but do not provide their plotted values.

B.4 Task Specific Results

  • The task-specific tables report cumulative reward and gain on each task's native scale, with cost for a complete run; results average five runs except for Codex.Bold marks the best value in each column, and the section also provides Pareto charts and learning curves for each task's best system.Source: §B.4 · pp. 21–22

B.5 Blind Spectrum Monitoring

  • Table 2 reports cumulative reward and gain across 90 Blind Spectrum Monitoring instances, while Figure 9 plots reward and gain against cost for system configurations.Figure 10 compares stateful and stateless mean reward per instance for ICL + GPT-5.4 on identical instances.Source: Table 2 · p. 22, Figure 9 · p. 22, Figure 10 · p. 23

B.6 Codebase Adaptation

  • Table 3 reports cumulative reward and gain across 19 Codebase Adaptation instances, while Figure 11 plots reward and gain against cost for system configurations.Figure 12 compares stateful and stateless mean reward per instance for ACE + GPT-5.4 on identical instances.Source: Table 3 · p. 23, Figure 11 · p. 24, Figure 12 · p. 24

B.7 Cohort Studies

  • Table 4 reports cumulative reward and gain across 20 Cohort Studies instances, while Figure 13 plots reward and gain against cost for system configurations.Figure 14 compares stateful and stateless mean reward per instance for ICL + GPT-5.4 on identical instances.Source: Table 4 · p. 25, Figure 13 · p. 25, Figure 14 · p. 26

B.8 Database Exploration

The appendix reports reward, gain, and cost comparisons and stateful-versus-stateless learning curves for three tasks, then describes a gain decomposition separating stability from plasticity and how variants are partitioned.

B.8 Database Exploration; B.9 Exploitable Poker; B.10 Sales Prediction

  • Database Exploration, Exploitable Poker, and Sales Prediction tables report cumulative reward and gain, standard errors across rollouts, and total API cost; figures show reward and gain versus cost.Each task also has a figure comparing the best system’s stateful learning curve with its stateless performance on identical instances.Source: Table 5 · p. 26, Figure 15 · p. 27, Table 6 · p. 28
  • For the best system in each task, learning-curve figures compare mean reward per instance in stateful mode with the same system run statelessly on identical instances.The best systems named are Claude Code + Sonnet 4.6 for Database Exploration and Exploitable Poker, and ICL Notepad + Claude Sonnet 4.6 for Sales Prediction.Source: Figure 16 · p. 27, Figure 18 · p. 29, Figure 20 · p. 30

C Stability–Plasticity Decomposition of Gain

  • The gain decomposition separates within-variant adaptation (plasticity) from reuse of older information across variants (stability), allowing equal overall gain to reflect different learning profiles.A highly plastic learner may adapt within variants but fail to carry learning forward, while a highly stable learner may retain structure but adapt more slowly.Source: C Stability–Plasticity Decomposition… · p. 30

C.1 Variant-based Partition

  • Variants are contiguous runs sharing properties or statistics, and they share latent structure predictive of future data streams.The partition defines the boundary and within-variant index sets for instances in a run.Source: §C.1 · pp. 30–31
  • Tasks are partitioned into boundary instances, where a new variant begins, and within-variant instances, which follow at least one round of in-variant feedback.The respective fractions of these instance types scale terms according to their representation in the task schedule.Source: §C.1 · p. 31

C.2 Decomposition

The decomposition defines stability and plasticity as additive shares of normalized gain, then interprets their signs and averages estimates across tasks.

C.2 Decomposition

  • The stability component uses stateful and stateless mean rewards restricted to a partition and scales the result by that partition’s share of task-level headroom.Component estimates are calculated per task and then averaged across tasks so tasks with more instances do not dominate.Source: §C.2 · p. 31
  • The stability and plasticity shares add to normalized gain, separating performance at variant boundaries from performance within variants.The method uses the same task-level headroom denominator as the overall gain measure.Source: §C.2 · p. 31
  • Positive stability indicates useful retention across variant boundaries, while negative stability can reflect misapplied learned information even if within-variant adaptation recovers performance.The two shares can therefore differ even when systems have similar overall gain.Source: §C.2 · p. 31

D Agent failure examples

The examples illustrate failures to carry useful information forward: a sales forecast responds inconsistently to recent feedback, while cohort estimates ignore relevant cross-study evidence.

D Agent failure examples

  • In Sales Prediction, the agent changed a five-year forecast from nearly flat to roughly 16% annual growth after interpreting feedback in the opposite direction from its earlier assessment.The passage describes immediate observations as dominating its prior view of long-run New York demand.Source: D Agent failure examples · pp. 31–32
  • The blind spectrum monitoring figure contrasts learners with 13.5% stability and 9% of learning headroom, illustrating different stability levels.The caption identifies Claude Opus 4.7 with ICL and GPT-5.4 with Codex, respectively.Source: Figure 21 · p. 32
  • A Round-1 forecast for a New York king bed frame remained essentially flat across five years after the agent recalibrated based on its reading of prior-round feedback.The forecast values were 230, 228, 227, 227, and 228 for 2035–2039.Source: D Agent failure examples · p. 32
  • In Cohort Studies, the agent dismissed 28 cohorts absent from Study 15’s variables and assigned all of them the same survival triplet, yielding an instance reward of 0.005.The agent had seen earlier study schemas and had relevant notepad content, but did not bring that evidence to the applicable cohorts.Source: D Agent failure examples · pp. 32–33
Loading 2606.05661v1…