PaperlayerView exact revision on arXiv

Source-linked AI summary

SCLATE: a Substrate for Continual-Learning Agent Training and Evaluation

Youngmok Jung, Sirajul Salekin, Henry Tran, Javier Movellan, Zhao Huang, Manjot Bilkhu

arXiv:2609.32391v2cs.AIcs.DCcs.LG

TL;DR

Existing benchmarks and training frameworks do not provide a shared way to schedule benchmark and agent events across multi-session lifecycles. SCLATE combines an open event scheduler, simulated clock, and rollout engine for unmodified agents; its evaluations find no universally best harness or memory, while post-training improves held-out MetaClaw accuracy by up to 11.8 points.

  • Problem

    Existing benchmarks and training frameworks do not coordinate benchmark scenarios with agent-side events across multi-session lifecycles, forcing custom scheduling loops for each benchmark-agent pair.

    Source: §1 · pp. 1–2

  • Method

    SCLATE uses adapters to register benchmark and agent events on an open scheduler with a hybrid simulated clock, and captures model calls through a proxy for post-training unmodified agents.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across ten models, no harness or memory is best for every model and added memory does not reliably beat native memory; held-out MetaClaw accuracy rose by up to 11.8 points after post-training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Continual-learning agents should be evaluated and trained with their model, harness, and memory together.

    Source: §1 · p. 2

Abstract

from arXiv · show

Continual-learning agents are systems of models, harnesses, and memory operating over long multi-session horizons. Evaluating and training them requires interleaving tasks with agent-side events such as session stop and start, crons, and memory consolidation. Yet existing benchmarks and training frameworks schedule only the benchmark's own events, leaving each benchmark and agent pair to build a custom scheduling loop. We present SCLATE, an execution substrate where benchmarks and unmodified agents each add their events to one open event scheduler through an adapter. A hybrid simulated clock runs these events on a shared timeline, flowing in real time while the agent works and skipping idle gaps, which compresses a month-long scenario into hours. SCLATE also serves as a rollout engine that runs any agent's harness and memory unmodified, recording the tokens and log probabilities of every model call through an in-container proxy. We port seven benchmarks to SCLATE and compare ten unmodified harness and memory configurations head to head on ten models. The comparison shows that an added memory system does not reliably beat the harness's native memory and that models differ widely in how they use the same harness and memory. We then post-train Qwen3.5-4B through unmodified harnesses and memory systems. The model learns to use both, reading 6.8x fewer file lines with a 16.7-point higher SWE-bench Verified pass rate, and writing richer memory records, while its held-out MetaClaw accuracy rises by up to 11.8 points.

1 Introduction

Continual-learning evaluation and training need to coordinate benchmark tasks with agent-side events across sessions, but existing systems leave this coordination unsupported. SCLATE provides a shared scheduler and rollout engine, and its evaluations show that agent performance depends on the model, harness, and memory together.

1 Introduction

  • Existing benchmarks and training frameworks lack a shared substrate for multi-session agent events, leaving researchers to build custom scheduling loops for each benchmark-agent pair.The cited limitations include missing temporal horizons or live loops, built-in assistants, or closed scheduling that excludes agent-side events.Source: §1 · pp. 1–2
  • SCLATE registers benchmark and agent events on one open scheduler and uses a hybrid clock that flows during work and skips idle intervals.Its adapters also support session stop and start, crons, and state save and restore; an in-container proxy records model-call tokens and log probabilities for post-training.Source: §1 · pp. 2–3
  • SCLATE is presented as an execution substrate for evaluating and training arbitrary continual-learning agents across lifecycles spanning days to weeks.The introduction describes continual-learning agents as systems combining memory, a harness, and model training.Source: §1 · pp. 1–2
  • Across ten models, no harness or memory is best for every model, added memory does not reliably beat native memory, and post-training improves held-out MetaClaw accuracy by up to 11.8 points.The comparison covers seven benchmarks and ten harness and memory configurations; the post-training result is across configurations.Source: §1 · p. 2

2 Related Work

Prior evaluation systems range from isolated memory tests to closed-agent or agent-agnostic benchmarks, while rollout frameworks lack some state and attribution capabilities needed for continual learning. SCLATE addresses these gaps with an open scheduler and stateful, component-aware rollout abstractions.

Continual-learning evaluation benchmarks

  • Evaluation benchmarks either score isolated dimensions without live agent loops, hardwire a built-in agent, or leave agent-side events outside the benchmark timeline.SCLATE instead decouples scheduling from the benchmark to coordinate benchmark and agent lifecycle events.Source: §2 · p. 3

Rollout engines for agent post-training

  • Existing rollout systems intercept model calls or standardize execution, while SCLATE additionally carries session state across container restarts and attributes calls to harness or memory components.Its abstractions manage rollout state across container lifecycles while remaining agnostic to benchmarks and training frameworks.Source: §2 · p. 3

3 The SCLATE Substrate

SCLATE provides a shared execution substrate for continual-learning evaluation and post-training, connecting benchmarks and unmodified agent harnesses through adapters. Its scheduler coordinates benchmark and agent events on a hybrid simulated clock, while container orchestration and an in-container proxy support isolated execution and trajectory capture.

3 The SCLATE Substrate

  • SCLATE uses three decoupled tiers—benchmarks and harnesses, host orchestration, and replicated sandboxes—to run evaluation and post-training without modifying harness source code.Adapters, configuration, and environment variables connect external harnesses; the scheduler coordinates events and the capture proxy records model calls.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Event Scheduling Loop and Hybrid Clock

  • At each tick, the scheduler snapshots time, world state, agent actions, and grading outcomes, selects due time- or condition-triggered events, and applies their effects.Effects can deliver prompts or notifications, mutate the world, or start, stop, save, or restore the agent and its memory.Source: §3.1 · p. 4
  • The hybrid clock flows in real time while agents or background jobs are busy, then skips idle intervals to the next event; Figure 3 reports jumps of up to 4.3× on Time scenarios.libfaketime exposes simulated time to dynamically linked harness and memory processes, allowing native crons to follow the scenario timeline.Source: §3.1 · p. 4, Figure 3 · p. 6

3.2 Pluggable Benchmark and Agent Adapters

  • Benchmark and Agent Adapters register benchmark progression and agent routines as event sources, enabling head-to-head evaluation of unmodified harnesses across benchmarks.The benchmark adapter supplies setup, event and grading interfaces; the agent adapter handles message delivery, session lifecycle, and state persistence.Source: §3.2 · pp. 4–5

3.3 Host Orchestration and Rollout Infrastructure

  • The host Orchestrator manages container lifecycles through a runtime interface, while the Rollout Engine gathers stateful trajectories for external training frameworks.Its runtime supports local Docker and managed cloud execution; the Rollout Engine exposes adapter save-and-restore primitives while leaving task selection and training steps to the external trainer.Source: §3.3 · p. 5
  • An in-container proxy records prompt tokens, completions, and log probabilities, assembles token-level trajectories, and lets trainer adapters distinguish agent turns from auxiliary calls.The host gathers trajectories with grading verdicts into training batches; Figure 3 also reports that scheduled environment events arrive on time across all three compared stacks.Source: §3.3 · p. 5, Figure 3 · p. 6

4 Experiments

Across seven benchmarks and ten models, the experiments validate SCLATE’s event scheduling and compare continual-learning configurations, then test post-training through unmodified harnesses and memory systems. Added memory does not reliably beat native memory, while post-training improves held-out MetaClaw accuracy and changes memory-writing and coding behavior.

4.1 Validating the Open Event Scheduler

  • SCLATE delivers all scheduled GAIA-2 events in the same order, with median delay of 0.9 seconds and maximum delay of 2.0 seconds.Across evaluations, it fires all 13,050 environment events and delivers 98.7% of checkable events at the scenario-specified delay.Source: §4.1 · p. 7
  • A 40-calendar-day MetaClaw arc finishes in a median of 5.9 real hours, a 162× median speed-up, while crons and memory consolidation follow simulated time.On GAIA-2 Time scenarios, clock jumps compress runs by up to 4.3×.Source: §4.1 · p. 7

4.2 Evaluating Continual-Learning Agents

  • Added memory does not reliably beat harness-native memory, and models vary widely in how they use the same harness and memory configurations.Figure 4 spans GAIA-2, AppWorld, LongMemEval-s, and PersonaMem-128k; the comparison covers ten configurations across ten models.Source: Figure 4 · p. 7, §4.2 · p. 8
  • The LongMemEval-s adapter turns static conversation logs into consecutive simulated-day sessions, requiring agents to carry earlier facts into the final-session question.The dataset content, questions, and ground-truth answers remain unchanged.Source: §4.2 · p. 8

4.3 Post-Training Continual-Learning Agents

  • Held-out MetaClaw accuracy rises by 7.6–11.8 percentage points in five of six configurations, with significant gains in three.The study trains on simulated days 1–20 and evaluates across all 346 rounds; Table 2 reports base and epoch-5 accuracy and improvements.Source: Table 2 · p. 8, §4.3 · p. 9
  • On Hermes with Hindsight, training increases facts containing a purpose clause from 1.1% to 19.6% and procedural rules from 23.6% to 40.6%.Figure 5 compares median metadata and statement length, as well as facts containing purpose clauses or procedural rules.Source: §4.3 · p. 9, Figure 5 · p. 9
  • On 42 matched coding tasks, the trained Claude Code model reads 6.8× fewer lines and raises SWE-bench Verified Pass@1 from 9.8% to 26.5%.It also reduces whole-file reads from 69.2% to 18.4% and context-limit hits from 97.6% to 31.0%.Source: Figure 6 · p. 9, §4.3 · p. 10
  • Across 73 SWE-Gym iterations, Claude Code rollout reward rises from 21.4% to 53.2%, while Codex stays near 69%.On SWE-bench Verified, Pass@1 rises from 28.8% to 33.6% under Codex.Source: Figure 6 · p. 9, §4.3 · p. 10

5 Conclusion

SCLATE uses an open event scheduler to place benchmark and agent events on one simulated timeline, avoiding custom scheduling loops for each pair. The conclusion reports that this supports long-horizon execution and post-training through unmodified harnesses and memory systems.

5 Conclusion

  • A shared simulated timeline lets benchmarks and agents use their own adapters instead of requiring a custom scheduling loop for every pair.The hybrid clock compresses 40 calendar days into a median of 5.9 hours while keeping events on schedule and in order.Source: §5 · p. 10

AI use statement

  • The authors report using generative AI for coding assistance and editing, and say they verified the paper’s implementations, designs, formulations, and claims.They state that they take full responsibility for the paper’s contents.Source: AI use statement · p. 10

A Comparison with Related Systems

B Interface Contract Specifications

SCLATE’s open scheduler polls benchmark and agent event sources, dispatches eligible effects, and advances simulated time according to whether the agent is active or idle. The section also describes scheduler measurements, time visibility inside agent processes, and a GAIA-2 grading adjustment.

B Interface Contract Specifications

  • The scheduler synthesizes a state snapshot, polls sources for candidate events, selects the earliest ready event, and dispatches its effects while recording completed fires.When active, simulated time advances with wall-clock time; when idle, it jumps to the next scheduled event.Source: B Interface Contract Specifications · pp. 13–14
  • The GAIA-2 adapter turns scenario tasks and oracle-DAG environment events into event sources, with state mutations and notifications attached to environment events.Figure 7 depicts the adapter; events without dependencies use a timed source keyed to scenario start and offset.Source: B Interface Contract Specifications · p. 13, Figure 7 · p. 15

C.1 Scheduler Measurements

  • Table 7 measures scheduler behavior across counted GAIA-2 cell-scenario runs and 100 MetaClaw runs.The measurements use scheduler traces, notification logs, and harness session transcripts.Source: §C.1 · p. 14

C.2 Simulated Time Inside Harness and Memory Processes; C.3 GAIA-2 Parity Grading

  • Figure 8 shows simulated time reaching MetaClaw harness and memory processes, while GAIA-2 parity grading waits for the final environment event and judges the whole trajectory.The parity protocol also reports interrupted turns as interrupted rather than as user-facing messages.Source: §C.2 · p. 14, §C.3 · pp. 14–15

C.4 Scheduler Walkthroughs on Real Trajectories

The walkthroughs align benchmark events, scheduler activity, and agent turns on a simulated-time axis, illustrating both real-time flow during work and jumps across idle intervals. They trace a multi-day MetaClaw lifecycle and three GAIA-2 scenarios with distinct event timings.

MetaClaw walkthrough

  • 40 calendar days of MetaClaw run in 11.0 real hours, with 70 jumps skipping 39.5 simulated days while sessions flow at real-time rate.On day 6, GBrain consolidates eight saved facts into three takes during a scheduler-registered nightly event.Source: §C.4 · p. 16, Figure 9 · p. 17

GAIA-2 Adaptability

  • The GAIA-2 Adaptability agent completes all 14 oracle actions, but only 3 of 15 harness-model cells pass the scenario.The agent acts on the friend’s reply and ignores a third-party email distractor; the clock flows throughout.Source: §C.4 · p. 16

GAIA-2 Time

  • In the GAIA-2 Time scenario, the scheduler jumps across idle gaps between four timed replies, and the judge matches all nine oracle actions.The scenario spans 334 simulated seconds in 201 real seconds.Source: §C.4 · pp. 16–17

GAIA-2 Adaptability notifications

  • 36.0% of GAIA-2 notifications reach Claude Code inside a running tool loop, where queued notifications are pulled into the active turn.The judge matches all eight oracle actions in the illustrated scenario.Source: §C.4 · p. 17

D.1 Benchmark Workload Specifications and Evaluation Protocol

The workload specifications describe how seven benchmarks are ported and evaluated under uniform execution semantics while preserving upstream task definitions and ground truth. The protocol also specifies benchmark subsets and matched comparisons across configurations.

Benchmark porting methodology

  • Benchmark ports preserve upstream task definitions and ground truth while adapting execution to simulated time and multi-session lifecycles.MetaClaw uses a simulated clock visible to processes; recall benchmarks schedule framework sessions on distinct simulated days, with agent restarts between days.Source: §D.1 · pp. 17–18

Dataset subsampling and split design

  • MetaClaw, GAIA-2, and AppWorld use their full public suites, while LongMemEval and PersonaMem each use the first 100 items of a specified split or tier.For post-training, the 293-task SkyRL-v0 SWE-Gym subset is used for training and all 500 SWE-bench Verified instances for downstream evaluation.Source: §D.1 · pp. 18–19

Faithful execution and scoring protocol

  • Canonical accuracy uses the latest faithful run per scenario, and reasoning effort is matched per model across driving harnesses.The protocol avoids optimistic best-of-k scoring and unconfounded inference-parameter differences.Source: §D.1 · p. 19

D.2 Evaluation Details and Tool Usage Analysis

The evaluation details report model differences in session-search behavior and token use across harnesses. The accompanying GAIA-2 table specifies the scope of the capability breakdown but supplies no outcome values in these passages.

D.2 Evaluation Details and Tool Usage Analysis

  • Gemini-3.7F calls session search substantially more often than other models, including under the unaugmented baseline.It inspects prior workspace transcripts and recovers procedural rules.Source: §D.2 · p. 19
  • 528M median tokens for Gemini-3.7F contrasts with 95–242M for other models; DS-V4.1 comes within 4.1 pp of Gemini’s median accuracy at 0.32× its tokens.Token consumption does not track accuracy; Hermes Native uses 2.6–6.4× Claude Code Native’s tokens for every API-served model because Hermes issues 2–4× more agent calls per round.Source: §D.2 · p. 20

D.3 Complete GAIA-2 Capability Breakdown

  • Table 11 lists Pass@1 (%) results for fifteen combinations of five models and three production harnesses across five GAIA-2 capability categories.The supplied passage specifies the table’s scope, not its cell values.Source: §D.3 · p. 20

D.4 AppWorld Evaluation Results

Table 12 reports AppWorld Pass@1 results across production harnesses for three models on the complete 585-task test suite.

D.4 AppWorld Evaluation Results

  • Table 12 reports Pass@1 (%) for Haiku-4.5, Sonnet-5, and DS-V4.1 across production harnesses on 585 AppWorld tasks.The suite includes 168 normal and 417 challenge tasks.Source: §D.4 · p. 20

E.1 Post-Training Infrastructure and Hyperparameters

Table 13 summarizes the configuration and infrastructure used for the SWE-Gym reinforcement learning runs reported in Section 4.3.

E.1 Post-Training Infrastructure and Hyperparameters

  • Table 13 summarizes the hyperparameters and infrastructure for the SWE-Gym reinforcement learning runs.The runs are reported in Section 4.3.Source: §E.1 · p. 20

E.2 MetaClaw OPD Post-Training Configuration

The MetaClaw OPD setup uses six harness-and-memory configurations and a teacher with privileged per-round feedback. A within-day split also yields higher held-out accuracy after training.

E.2 MetaClaw OPD Post-Training Configuration

  • The six OPD runs each use a different harness-and-memory combination.The configuration is summarized in Table 14.Source: §E.2 · p. 20
  • With one sample per prompt, group size one, and no task reward, the logged advantage is the negative reverse-KL log-ratio.The smoothed sampled reverse-KL estimates remain between 0.04 and 0.16 throughout training.Source: §E.2 · p. 20
  • The teacher receives per-round context that the student never sees, including the benchmark’s corrective feedback for each of 346 rounds.The context covers 30 simulated days; its strings range from 64 to 1,061 characters, with a median of 222.Source: §E.2 · p. 20

E.3 Within-Day Partition

  • The within-day evaluation trains on each day’s first two-thirds of rounds and tests on its final third.The held-out tail contains 104 of the 346 rounds, with assignment fixed in advance.Source: §E.3 · pp. 20–21
  • Held-out accuracy rises from 27.9% to 48.1% under the within-day split for Hermes with Mem0.Training-slice accuracy rises from 21.5% to 42.1%.Source: §E.3 · p. 21

F Memory-Write Analysis Controls and Additional Measurements

The memory-write analysis controls the backend and extraction setup, then examines how training changes procedural-rule content and record structure. The supplementary measurements also describe token accounting and the captured training trajectory.

Experimental controls

  • Both memory-write arms use the same Hindsight daemon, extraction prompt, container image, and launch command; only the served model and upstream URLs differ.The facts use the backend’s own pipe-delimited template rather than a model-invented structure.Source: F Memory-Write Analysis Controls and… · p. 21

Length controls

Distribution and per-rule detail

Multi-entity records

Token usage

  • Across 38 memory-backend cells with complete usage records, memory-system calls account for a median 4.2% of tokens and at most 13.8%.The table totals uncached input, cache reads, cache writes, and output; self-hosted endpoint usage fields are omitted where unavailable.Source: Table 10 · p. 26

Verbatim examples

  • Matched examples show trained records naming artifacts, attributing them to the agent, and stating their purpose, unlike the untrained records’ instruction restatements.The examples come from days selected by round gains; the round tallies are not tied to the record-producing round.Source: F Memory-Write Analysis Controls and… · p. 21, Figure 19 · p. 31

Related papers

Loading 2609.32391v2…