PaperlayerView exact revision on arXiv

Source-linked AI summary

ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience?

Haizhong Zheng, Yizhuo Di, Ranajoy Sadhukhan, Shuowei Jin, Beidi Chen

arXiv:2610.07792v1cs.LGcs.AI

TL;DR

Agents need to infer and revise hidden environment knowledge from experience as real-world policies change, but existing benchmarks cover this challenge only partially. ServeLearnBench evaluates learning across evolving streams and finds substantial gaps between task capability and learning from experience, alongside adaptation costs and performance degradation.

  • Problem

    Existing benchmarks often provide target knowledge, assume static environments, or offer limited scale and knowledge diversity for evaluating discovery and revision of hidden knowledge.

    Source: §1 · p. 2

  • Method

    ServeLearnBench formalizes evolving-environment streaming datasets and tests agents across changing hidden policies using interaction and outcome feedback.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Substantial gaps remain in acquiring and revising hidden knowledge despite task capability with disclosed policies, while continual adaptation costs more and can degrade already-correct behavior.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    ServeLearnBench provides a systematic testbed for diagnosing continual-learning challenges and tracking progress toward agents that learn reliably from deployment experience.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language model agents are increasingly deployed to perform complex tasks in real-world environments. However, the knowledge required for correct behavior in these environments is often implicit, undisclosed, and subject to change over time. Recent continual-learning harnesses seek to address this challenge by enabling agents to improve from serving experience. Yet the effectiveness and limitations of these methods are not yet well characterized. Existing benchmarks provide only partial coverage: some explicitly provide the target knowledge, others assume a static environment, and those that support continual adaptation remain limited in scale and knowledge diversity. To enable systematic evaluation, we formalize an evolving-environment streaming dataset (EESD), in which agents must infer, apply, and revise latent environment knowledge from interaction and outcome feedback as hidden policies evolve, and introduce ServeLearnBench, spanning retail support, banking, and sales-pitch generation with 53 environment windows and 7,718 tasks. We evaluate five learning harnesses (RAG, Mem0, SkillOpt, Continual Harness, and Prime) across six models (GPT-5.6 Terra, Opus 5, Kimi K3, GLM-5.3, DeepSeek V4.1 Flash, and GLM-5.3 Flash), covering 28 model-harness pairs and 252 learning runs. Our evaluation reveals three main findings: a substantial gap remains between task capability and learning from experience; continual adaptation is costly and can degrade already-correct behavior; and insufficient exploration emerges as a key bottleneck to effective adaptation. Overall, ServeLearnBench provides a controlled testbed for diagnosing these limitations and tracking progress toward agents that continually and reliably improve through serving experience.

1 Introduction

Agents must infer and revise hidden environment knowledge from serving experience as real-world policies change, but existing benchmarks cover only parts of this problem. ServeLearnBench evaluates that capability in evolving task streams and finds persistent learning gaps, adaptation costs, and an exploration bottleneck.

1 Introduction

  • Existing benchmarks often reveal target knowledge, assume static environments, or cover evolving environments at limited scale and knowledge diversity.This leaves acquisition and revision of hidden knowledge, adaptation costs, and mechanisms of effective continual learning difficult to measure systematically.Source: §1 · p. 2
  • ServeLearnBench formalizes evolving-environment streaming datasets, where agents infer latent knowledge from interaction and outcome feedback as hidden policies change.The benchmark contains 53 environment windows, 4,508 serving tasks, and 3,210 held-out test tasks across retail support, banking case review, and sales-pitch generation.Source: §1 · p. 2, Figure 2 · p. 3
  • Across the 28 model–harness pairs, the best Retail L3 score is 59.6%, while the median is 22.0%, despite Oracle averaging 95.4% on Hidden cases.The gap indicates that task capability with disclosed policies does not translate into reliable learning of hidden policies.Source: §1 · p. 2
  • Hidden-policy inference costs 4–101× as much as following a disclosed policy, and adaptation can hurt tasks requiring no learning.On Kimi K3, Prime costs $2,479 versus $102 for Oracle.Source: §1 · p. 2
  • Harnesses that vary candidate behaviors more achieve higher overall reward, and exploration during unsuccessful adaptation is strongly associated with final performance.The paper identifies effective exploration as an important design target, while the reported relationship is an association.Source: §1 · p. 3

2 Related Work

Prior work studies interactive policy following, feedback streams, persistent memory, changing environments, and self-evolving agents, but these benchmark families emphasize different capabilities. ServeLearnBench targets their partial intersection: hidden evolving state, outcome feedback without corrections, and evaluation of learning gains alongside collateral effects.

Memory and continual-learning benchmarks

  • Interactive-agent and feedback-stream benchmarks assess policy following, tool use, language feedback, or improvement across tasks, while lifelong-agent suites study acquisition, reuse, composition, and retention.StateMemBench focuses on tracking revised facts and distinguishing current from superseded state.Source: §2 · p. 3
  • Dynamic-environment benchmarks vary in what changes, including interactive environments, drifting versus stable knowledge, and adaptive practice over otherwise static environments.These benchmark families provide complementary forms of environmental change rather than sharing one limitation.Source: §2 · p. 3
  • What remains only partially combined is evaluator-controlled change with hidden operative state, outcome-only feedback, potentially stale or reactivated knowledge, and matched controls for collateral effects.ServeLearnBench addresses this intersection with evolving-environment streams, Blind/Oracle references, and Hidden versus Fully Specified slices.Source: §2 · p. 3

Continual learning for self-evolving agents

  • Self-evolving agents update different state, from retained evidence and distilled strategies to predictive rules, prompts, skills, harness state, code, or search strategies.ServeLearnBench evaluates inference of hidden knowledge from serving outcomes, revision after changes, and preservation of behavior on Fully Specified cases.Source: §2 · p. 3

3 ServeLearnBench

ServeLearnBench defines an evolving-environment streaming dataset and instantiates it across retail, banking, and pitch-generation tasks. Its windows test transfer to new instances under changing hidden policies, while task types and difficulty tiers vary the knowledge and evolution agents must handle.

3.1 Evolving-Environment Streaming Dataset (EESD)

  • An evolving-environment streaming dataset is a chronological task stream in which a fixed learner-visible interface faces an evaluator-private environment that evolves over time.The formulation tests acquisition, application, and revision of environment knowledge from prior interactions.Source: §3.1 · p. 4
  • Environment windows have latent states that may introduce, revise, or reactivate policies, and neither the active state nor its changes are disclosed to the harness.Serving and test tasks use disjoint instances under the same active components, so tests assess transfer rather than memorization; Hidden-Dependent tasks require latent-state knowledge, unlike Fully Specified tasks.Source: §3.1 · p. 4

3.2 ServeLearnBench: Task Creation

  • Retail tasks test multi-turn order changes and returns, Banking tasks test case-review decisions, and Pitch tasks require 40–80-word sales pitches tailored to latent customer preferences.Retail correctness depends on the intended database state or refusal; Banking policies govern case decisions, while Pitch rewards preference coverage and penalizes undesirable attributes and unsupported claims.Source: §3.2 · pp. 4–5
  • The three difficulty tiers progress from unseen-state acquisition to policy revision and composition, then to non-monotonic changes including reversals, removals, and re-entry.Pitch profiles likewise progress from non-repeating to recurring and then cycling back to previously seen profiles.Source: §3.2 · p. 5
  • Blind measures performance without serving experience, while Oracle receives each window’s active hidden policy to remove hidden-state inference from the task.ServeLearnBench uses domain-specific rewards for database transitions or refusals, Banking verdicts and states, and Pitch preference coverage and penalties.Source: §3.2 · p. 5

4 Evaluation

The evaluation crosses models and learning harnesses over the benchmark’s scenario–difficulty settings, then compares performance and resource costs. Oracle substantially outperforms Blind on average, while Prime and Continual Harness generally lead the learning harnesses; higher reward can require greater cost and adaptation may harm Fully Specified performance.

4 Evaluation

  • The benchmark evaluation organizes analysis around learnable headroom, how much streaming learners capture, and whether adaptation harms Fully Specified cases.The performance–resource analysis considers reward, token use, and API cost; token and monetary efficiency can produce different rankings.Source: §4 · p. 5, §4.3 · p. 7

4.1 Experimental Setting

  • The evaluation compares six models with five learning harnesses and Blind and Oracle references across scenario–difficulty settings, with 28 of 30 possible model–harness pairs evaluated.The same task streams and scoring are used while harness-native runtime, update schedule, replay access, learning computation, and state capacity are preserved.Source: §4.1 · p. 5, §4 · p. 5

4.2 Main Evaluation Results

  • Prime and Continual Harness generally outperform the other learning harnesses on Hidden cases, but remain below Oracle; Oracle averages 95.4 versus Blind’s 14.1.Across the four open-weight models, Prime and CH average 61.7 and 57.6, compared with 38.3–43.4 for the other three; CH leads on GLM-5.3 while Prime leads on the other three models.Source: §4.2 · pp. 6–7, Table 2 · p. 6, Figure 3 · p. 7

4.3 Cost Comparison

  • Prime achieves the highest harness-level Hidden reward at 59.0 for $0.195 per task, while CH reaches 55.1 for $0.083.Across models, GLM-5.3 and Kimi K3 reach Hidden rewards of 49.8 and 49.2 at 14–16× GLM-5.3 Flash’s $0.010 per-task cost, which yields 42.0.Source: §4.3 · p. 7

5 Analysis

Section 5 analyzes adaptation speed, effects on already-correct cases, and whether exploration and post-success reuse relate to reward. Across these analyses, harnesses differ in how quickly they learn, how adaptation affects Fully Specified accuracy, and how well they explore and retain successful behavior.

5.1 How Fast Do Harnesses Learn a New Policy?

  • 54.9 and 49.9 AULC10 scores for Prime and CH in Banking, and 37.7 and 36.2 in Retail, make them the fastest-learning harnesses in both domains.After five examples of a new Banking policy, both answer roughly 60% of subsequent tickets correctly; SkillOpt is slowest, with AULC10 scores of 18.2 in Banking and 11.1 in Retail.Source: Figure 5 · p. 8, §5.1 · p. 8

5.2 Does Adaptation Harm Fully Specified Cases?

  • 98.9 for Blind versus 89.5–95.5 for every learning harness in Retail shows that adaptation lowers Fully Specified accuracy there.In Banking, accuracy is 91.7 for Blind versus 83.6–86.2 for all but Prime (95.2) and Mem0 (92.7); on GLM-5.3 Flash, CH raises Hidden performance from 14.6 to 49.7 while Fully Specified performance falls from 96.7 to 82.8, whereas Prime reaches 61.1 Hidden and retains 95.2 Fully Specified.Source: §5.2 · p. 8

5.3 Does Exploration Predict Adaptation?

  • 1.16, 1.02, 0.88, 0.65, and 0.51 exploration AUCs for Prime, CH, RAG, Mem0, and SkillOpt exactly match their mean Hidden reward ordering (ρ = 1.00).The association persists at harness–domain–tier level (ρ = 0.75); harness exploration ranges from 0.51–1.16, compared with 0.77–0.92 across models.Source: Figure 7 · p. 9, §5.3 · p. 9
  • 78% and 76% AUCKA for Prime and CH exceed SkillOpt’s 68%, while Mem0 and RAG reach only 41% and 38%.Post-success curves are largely flat, and AUCKA correlates with Hidden reward at only 0.60; SkillOpt reuses correct behavior well once found but struggles to discover it, while RAG often finds it but reuses it unreliably.Source: Figure 8 · p. 9, §5.3 · p. 9

6 Conclusion

ServeLearnBench tests acquisition and revision of hidden environment knowledge across evolving environments. The evaluation finds substantial remaining gaps, computational cost, and possible degradation of already-correct behavior, while identifying exploration and knowledge acquisition as distinct bottlenecks.

6 Conclusion

  • Across three domains and nine scenarios, the evaluation finds gaps in acquiring and revising hidden knowledge despite strong task performance when policies are disclosed.Continual adaptation also carries computational cost and can degrade already-correct behavior; agents must discover useful behavior and convert success into reusable knowledge.Source: §6 · p. 10

B.2 Learning Harnesses

  • The evaluation holds model and reasoning effort constant within each model block and uses shared task streams, held-out tests, and per-task budgets, but not equal information or resource budgets across harnesses.Comparisons characterize complete systems under their operating conditions rather than isolating harness effects under identical budgets.Source: §B.2 · p. 15
  • Prime uses its own sandboxed code-agent runtime, while RAG, Mem0, SkillOpt, and CH share a ReAct loop, so CH–Prime differences include runtime as well as state-layer effects.The shared loop allows one tool call per turn, while Prime may issue several calls per turn.Source: §B.2 · p. 15
  • Harnesses learn through different update schedules: Prime writes post-score memory, CH refines on a cadence, RAG indexes blocks of twelve cases, and SkillOpt optimizes in twelve-ticket blocks.SkillOpt’s selection replays rerun served tasks, about 0.85 additional executions per served ticket, a simulator-enabled procedure that does not fit live-ticket deployment such as Pitch.Source: §B.2 · p. 15
  • Prime searches past transcripts, CH bounds its Refiner’s window, and SkillOpt limits edits to one skill document; these unequal capacities are part of each method.Learning-side calls are booked as maintenance, while Prime’s learning share is recovered from per-ticket session records.Source: §B.2 · p. 15

B.3 Evaluation Protocol

  • Learners receive task-visible information and serving outcomes, may update persistent state from experience, and receive no feedback on test tasks.The objective is improving Hidden-Dependent performance while preserving Fully Specified behavior already supported by visible information.Source: §B.3 · pp. 15–16
  • Model-visible inputs exclude latent policies, corrected actions, evaluator explanations, and slice labels, though harnesses may use window segmentation for scheduling or replay.The release gate audits prompts and metadata against these content restrictions.Source: §B.3 · p. 16
  • Each environment window has serving tasks followed by test tasks under the same latent state; test probes run from frozen learner state and are excluded from all learner updates and selection.Serving outcomes may inform updates, while subsequent replay scores are not included in reported serving or test scores.Source: §B.3 · p. 16

B.4 Data Processing and Reporting

The reporting appendix defines how benchmark rewards and costs are aggregated, and documents the benchmark specification and limits on interpreting run-level results.

B.4 Data Processing and Reporting

  • Table 2 reports held-out test aggregates from current completed cells, with averages computed from published one-decimal setting scores.Superseded archived runs are excluded, and incomplete sets of constituent settings appear as a dash.Source: §B.4 · p. 16
  • Cross-setting reward summaries average equally weighted model–harness–setting cells, while per-task costs divide total dollars by total tasks.Retail and Banking rewards are scaled to 0–100, as is the Pitch rubric score; the per-setting results remain available in Table 2.Source: §B.4 · p. 16
  • Each result cell summarizes one run, so the table implies no standard errors or confidence intervals.Pitch L1 has only 6–8 effective attribute combinations per 24-item window, so its item count is not independent statistical replication.Source: §B.4 · p. 16
  • Table 3 reports per-task cost and reward, aggregating 45 lanes per model column and 36 per harness column.Its cost includes serving-task learning work; test cost is measured under the frozen learned state.Source: Table 3 · p. 17

C Detailed Benchmark Specification

  • The detailed specification covers learner-facing interfaces, tier and window design, evaluator-only hidden-state schedules, and task-success verification.Latent-state descriptions are evaluator metadata, not supplied to learning harnesses.Source: C Detailed Benchmark Specification · p. 17

C.1 Task and Scenario Interfaces

The benchmark combines three task domains with tiered, windowed environments and latent rules or preferences that change over time. Its specification separates task interfaces from the activation schedules that govern each scenario.

C.1 Task and Scenario Interfaces

  • Retail and banking require tool-mediated decisions verified from the final environment state, while Pitch uses constrained natural-language responses scored by a fixed rubric.Table 4 distinguishes the surface task from the latent object that changes.Source: §C.1 · p. 17, Table 4 · p. 17

C.2 Tier Semantics and Window Sizes

  • Tier labels describe temporal and compositional demands, not a monotonic rise in surface-form complexity; serving and test use disjoint instances of the same active components.Table 6 lists serving and test counts for each released window, covering 53 windows and 4,508 serving plus 3,210 test tasks.Source: §C.2 · pp. 17–18, Table 6 · p. 18

C.3 Retail Hidden Rules and Activation Schedule

  • Retail L1/L2/L3 use 7/14/12 categorical hidden rules, with tables separating atomic rule meanings from each window’s rule activation.Rules cover payment, products, destinations, refunds, order composition, and exceptions; block rules override permissions, while permit rules introduce exceptions to restrictions.Source: §C.3 · pp. 18–19, Table 7 · p. 18, Table 8 · p. 20

C.4 Banking Hidden Rules and Activation Schedule

  • Banking L1/L2/L3 use 5/9/11 rule-value states across 4/6/6 policy dimensions, with tables defining atomic rules and their window-level activation.The activation table marks rules active, inactive, or unavailable by window.Source: §C.4 · p. 21, Table 9 · p. 21, Table 10 · p. 21

C.5 Pitch Latent Preferences and Activation Schedule

  • Each Pitch window activates one complete customer preference profile; the benchmark’s dictionaries and activation matrices make profile meaning and temporal re-entry distinct.L1 uses four non-repeating profiles across four windows; L2 and L3 each use four profiles with repeats across six and seven windows, respectively.Source: §C.5 · p. 22, Table 11 · p. 22, Table 12 · p. 22

C.6 Task Success Verification

The verifier checks Retail and Banking outcomes deterministically, while Pitch uses structured LLM judgments mapped to a numerical score. The appendix also covers additional evaluation analyses.

C.6 Task Success Verification

  • Retail and Banking use a rule-based, outcome-only verifier that compares the agent’s terminal database with a reference state replayed from ground-truth tool actions.Canonicalized database hashes determine whether the terminal outcomes match, without requiring a particular sequence of intermediate actions.Source: §C.6 · p. 22
  • A correct final answer cannot compensate for an incorrect database transition, and a correct state cannot compensate for a missing or inconsistent finish declaration.The task requires one structured finish action, with typed answers matched to the expected value format.Source: §C.6 · pp. 22–23
  • Pitch scoring uses a fixed GLM-5.3 Flash judge to classify attribute coverage, contradictions, and unsupported claims before deterministic code computes the rating.The weighted positive-coverage score is adjusted for negative and red-line attributes, contradictions, unsupported claims, and requested pitch length.Source: §C.6 · p. 23

D Additional Evaluation

  • The additional evaluation appendix covers policy-level learning speed, behavior after policy withdrawal, answer diversity, serving-stream costs, training-based learning attempts, and Fully Specified reference performance.Source: D Additional Evaluation · p. 23

D.1 Learning-Speed Analysis

The analysis examines policy-level learning and what happens when learned policies change or are withdrawn. It finds that dropping a learned policy is about as fast as learning it, while multi-rule tickets complicate learning-speed measurement.

D.1 Learning-Speed Analysis

  • The analysis covers 68 Banking and Retail policies, including states present from the start, tightened, relaxed, re-scoped, introduced, or removed.The count is 31 Banking and 37 Retail policies across the three tiers.Source: §D.1 · p. 23
  • Nearly half of Hidden serving tickets in Banking L2 and L3 are decided by two rules at once, and measuring only single-rule tickets creates spurious Banking curve dips.The multi-rule counts are 82 of 172 in Banking L2 and 101 of 198 in Banking L3; counting such tickets for both exposure and measurement preserves harness rankings.Source: §D.1 · p. 24

D.2 Learning a Policy versus Dropping It

  • The comparison retains only pairs that learned the policy, requiring at least 60% correctness on its final five tickets to avoid an artificially easy dropping score.Both sequences use the same model, harness, and policy; dropping tickets are correct under the manual and wrong under the ended policy.Source: §D.2 · p. 24
  • Dropping a learned policy is about as fast as learning it; after seven to ten tickets, both curves end 8–14 points higher on dropping than learning.Prime starts at 20% when dropping versus 7% when learning, while Mem0 learns faster than it drops on tickets four to six; the Mem0, RAG, and SkillOpt panels use 20–24 pairs.Source: §D.2 · p. 24, Figure 9 · p. 24
  • Figure 10 plots distinct-answer counts after successive Hidden tickets and exploration AUC against average Hidden-case reward, including Spearman correlations and least-squares fits.Source: Figure 10 · p. 25

D.3 Exploration by Domain

Across harnesses, exploration diversity tracks Hidden reward, with the relationship examined at harness, domain, tier, and model–harness levels. The reported correlations vary by grouping and domain, and the analysis notes limits in which policies contribute to the curves.

D.3 Exploration by Domain

  • Curves include only policies with at least K = 8 tickets, and policies longer than K contribute only their first K tickets.The horizon trades off reading more of each curve against retaining more policies.Source: §D.3 · p. 25
  • For every K from 4 to 12, harness exploration-AUC ranking was Prime, CH, RAG, Mem0, SkillOpt, with rank correlation 1.00 to average Hidden reward.SkillOpt’s once-per-12-ticket updates make its within-block answer diversity a lower bound on what per-ticket updating would show.Source: §D.3 · p. 25
  • Retail exploration AUC and average Hidden reward had ρ = 1.00, while Banking had ρ = 0.90; Mem0 and SkillOpt swapped order in Retail.In Banking, the AUCs were 1.50, 1.44, 1.35, 1.20, and 0.88 for Prime, CH, RAG, Mem0, and SkillOpt, respectively.Source: §D.3 · p. 25
  • Across 30 harness–domain–tier cells, rank correlation was 0.75 overall, 0.54 in Retail, and 0.69 in Banking, with the same direction in every subset.Tier reward differences and domain answer-space differences add variance to the finer-grained comparison.Source: §D.3 · p. 25
  • At the model–harness level, AUC correlated with Hidden reward at ρ = 0.92 in Retail and 0.57 in Banking, where fast-learning cells stop varying answers early.A distinct-answer-per-ticket variant before the first correct answer reached ρ = 0.94 in Retail and 0.73 in Banking; its median within-model correlations were 0.90 and 0.75.Source: §D.3 · p. 26

D.4 Cost per Phase and Learning Cost Breakdown

The cost analysis separates stream-time serving and learning work from inference on held-out tasks under a frozen learned state. It finds cost and reward trade-offs across phases and substantial harness-specific variation in the share of computation spent learning.

D.4 Cost per Phase and Learning Cost Breakdown

  • The harness ranking can differ by phase: SkillOpt and Mem0 spend most serving budget on learning, while RAG’s held-out task costs $0.083 versus $0.066 per serving task as retrieved exemplars lengthen its prompt.SkillOpt and Mem0 held-out tasks cost less than half their serving tasks.Source: §D.4 · p. 26
  • Prime consumes the most tokens per task on every model, because each ticket opens a fresh session that rereads harness state over 20–29 model calls.Prime is also the most expensive harness per task on every model, ranging from $0.026 on GLM-5.3 Flash to $0.46 on GLM-5.3.Source: §D.4 · pp. 26–27
  • Prime costs $0.232 per serving task and $0.142 per held-out task, with the highest Hidden test reward (61.7); CH reaches 57.7 at 52% of Prime’s test cost ($0.074).Serving cost includes learning work and is paired with serving-case reward; held-out test cost is inference under the frozen learned state.Source: §D.4 · p. 26, Figure 12 · p. 27
  • Learning accounts for 56–62% of SkillOpt’s API cost, 48–76% of Mem0’s, 14–31% of CH’s, and 30–40% of Prime’s.CH’s learning share is 5–8% of tokens because serving turns resend prompt, memory, and skill overviews, mostly from cache.Source: §D.4 · p. 26, Figure 13 · p. 28

D.5 Training-Based Methods

The appendix reports preliminary difficulty obtaining successful trajectories for training-based methods, then qualifies how Fully Specified reference performance should be interpreted.

D.5 Training-Based Methods

  • In preliminary trials, 14B and 32B models struggled to infer hidden knowledge and complete tasks, making successful self-generated trajectories difficult to obtain even with strong harnesses.This limits correct demonstrations for success-filtered SFT and reward contrast for GRPO under sparse outcome rewards; uniformly failed groups provide no preference among trajectories.Source: §D.5 · p. 28

D.6 Interpreting Fully Specified Reference Performance

  • Fully Specified tasks have sufficient visible specifications, but models may not trust or follow them; unsupported restrictions can lower Blind performance even when task-specific evidence is absent.The authors therefore interpret changes relative to Blind as differences from the no-learning harness, not pure learning-induced damage.Source: §D.6 · p. 28

Related papers

Loading 2610.07792v1…