PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning What to Remember: Resource-Efficient Self-Improvement for Lifelong Agents

Guannan Lai, Zi-Yang Bai, Jun-Peng Jiang, Han-Jia Ye

arXiv:2610.26453v2cs.AI

TL;DR

Self-improving agents must learn from experience despite jointly constrained generation, interaction, and memory resources, where relevance alone may not indicate an experience’s downstream utility. TokenWorth learns query-dependent memory worth from sparse execution feedback to guide retrieval and bounded memory management. Across three LifelongAgentBench environments and local and API-based deployments, it achieves the highest task success rates while using the fewest output tokens among evaluated methods.

  • Problem

    Under joint generation, interaction, and memory constraints, accumulating and reusing experience does not necessarily improve decisions, and semantic relevance may not indicate downstream utility.

    Source: §1 · pp. 1–2

  • Method

    TokenWorth learns query-dependent memory worth from sparse execution feedback and uses it to guide retrieval and bounded memory management without updating the backbone.

    Source: §1 · p. 2

  • Results

    Across three LifelongAgentBench environments and local and API-based deployments, TokenWorth achieves the highest task success rates among evaluated methods while using the fewest output tokens.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Under resource constraints, effective self-improvement depends on learning which experiences are worth remembering, not simply accumulating more experience.

    Source: §6 · p. 9

    Feedback reflects downstream task success rather than the counterfactual causal contribution of a selected memory, since TokenWorth does not re-execute tasks with alternative memories.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Self-improving agents aim to continuously improve their behavior by learning from experience after deployment. However, accumulating and reusing more experience does not necessarily lead to better decisions, particularly when generation, interaction, and memory resources are jointly constrained. Under such stringent constraints, we find that several existing lifelong-memory methods provide limited benefits over simple retrieval or even no-memory agents. This reveals a fundamental challenge for resource-efficient self-improvement: agents must learn not only which past experiences are relevant, but which are actually worth reusing to improve future decisions. When memory capacity and opportunities for reasoning and recovery are limited, retrieving an ineffective experience can consume scarce resources and hinder downstream execution. To address this challenge, we propose TokenWorth, a lightweight feedback-driven memory adaptation framework that learns query-dependent memory worth from sparse execution feedback. Without updating the backbone model, TokenWorth uses the learned worth to guide both experience retrieval and bounded memory management, enabling the agent to continually adapt how it reuses accumulated experience. Across three LifelongAgentBench environments and both local and API-based deployments, TokenWorth achieves the highest task success rate while using the fewest output tokens among the evaluated methods under the specified resource budgets. These results suggest that effective self-improvement under resource constraints depends not on accumulating more experience, but on learning what is worth remembering. Our code is publicly available at https://github.com/AIGNLAI/TokenWorth.

1 Introduction

Under joint generation, interaction, and memory constraints, accumulating experience alone may not improve decisions. TokenWorth learns query-dependent memory worth from execution feedback, and across six environment–deployment combinations it achieves the highest task success rate while using the fewest output tokens among evaluated methods.

1 Introduction

  • Under stringent joint resource constraints, several memory approaches offer limited gains over simple retrieval or no memory, and some underperform in certain environments.Different memory policies can produce markedly different performance under the same budget.Source: §1 · pp. 1–2
  • The paper studies agents adapting from experience under bounded generation, interaction, and memory resources, where relevant experiences may still lack downstream utility.When reasoning and recovery opportunities are limited, ineffective retrieval consumes scarce resources and useful experience may be unavailable if not retained or selected.Source: §1 · p. 2
  • TokenWorth learns query-dependent memory worth from sparse execution feedback and uses it for retrieval and bounded memory-bank management without backbone updates or additional language-model calls.Its scorer combines an interpretable online base scorer with a residual Recursive Feature Machine for query-specific corrections.Source: §1 · p. 2
  • Across all six environment–deployment combinations, TokenWorth achieves the highest task success rate while using the fewest output tokens among evaluated methods.The evaluation spans three LifelongAgentBench environments and both local and API-based deployments.Source: §1 · p. 2

2 Related Work

Prior work studies experience accumulation and reuse, resource-aware adaptation, and benchmarks for continual agent learning. This paper focuses on learning which experiences are worth reusing under jointly constrained generation, interaction, and memory resources.

Experience-Driven Self-Improvement in Language Agents

  • Prior systems use trajectories, reflections, reusable skills, and explicit memory mechanisms for experience-driven adaptation, while benchmarks evaluate continual agent learning.Examples include Reflexion [19], ExpeL [33], Voyager [24], SocioDojo, MemoryBank [37], Mem0 [5], MemRL [32], MemSkill [30], and AutoSkill [27].Source: §2 · p. 3

Resource-Constrained Agent Adaptation

  • Resource-aware adaptation work considers inference budgets, bounded on-device memory, runtime memory routing, or memory valuation, whereas this paper studies their joint constraints.TokenWorth learns query-dependent memory worth from online execution feedback to guide both retrieval and bounded memory management without backbone updates.Source: §2 · p. 3

3 Resource-Constrained Lifelong Agents

The paper formalizes lifelong agents as processing sequential tasks while adapting through an online memory policy under joint resource limits. The policy selects experiences for retrieval and updates memory after task outcomes.

3 Resource-Constrained Lifelong Agents

  • A lifelong agent processes an online task stream, retrieves past experiences from its memory bank, and receives a task-level binary reward after interaction.The backbone is frozen; adaptation occurs through external memory and its management policy.Source: §3 · p. 3
  • The joint budget constrains generation, interaction rounds, memory-bank capacity, retrieval count, and the length of injected memories.Generation bounds include output tokens from auxiliary language-model calls such as memory construction, reflection, or skill extraction.Source: §3 · pp. 3–4
  • The online memory policy determines both which experiences to retrieve and how to update the bank after outcomes, with its lifelong task performance evaluated under a fixed resource budget.The feasible policy class consists of online memory policies satisfying the resource constraints.Source: §3 · p. 4

4 TokenWorth: Feedback-Driven Memory Adaptation

TokenWorth learns query-dependent memory worth from execution feedback, then uses it to guide experience selection and bounded memory management. Its linear scorer and residual learner adapt at different rates to balance stable patterns with query-specific usefulness.

4.1 Overview

  • TokenWorth scores recalled experiences for the current query, selects the highest-valued candidate for execution, and updates worth from the resulting reward.Learned worth also guides memory-bank management, without changing the backbone.Source: §4.1 · p. 4

4.2 Learning Query-Dependent Memory Worth

  • The base scorer and residual learner update from execution feedback for the selected query–memory pair, but this supervision is not a counterfactual estimate of that memory's causal contribution.Tasks run once, and failures may arise from factors beyond the selected memory.Source: §4.2 · pp. 4, 6
  • A prior-regularized linear scorer captures stable, interpretable utility patterns, while an RFM residual models nonlinear, query-specific deviations.The linear component updates conservatively around an informative prior; the residual complements it with more flexible corrections.Source: §4.2 · pp. 4–6
  • Candidates are recalled using lightweight textual relevance and represented by similarity, historical utility, reliability, and recency features.The richer residual representation combines hashed text-similarity features with each memory's empirical history.Source: §4.2 · p. 5

4.3 Two-Timescale RFM Adaptation

  • Fast KRR coefficient updates track recent residual structure, while slower AGOP-based metric updates adapt feature geometry more conservatively.Together with the prior-regularized linear scorer, the two timescales aim to balance responsiveness and stability under sparse feedback.Source: §4.3 · p. 6

4.4 Worth-Guided Memory Lifecycle

  • TokenWorth averages memory worth over recent probe queries for retention, admits or replaces candidates under coverage–quality criteria, and evicts the lowest-worth memory when capacity is full.Query-dependent worth guides immediate retrieval; probe-averaged worth guides persistent storage, within the same bounded memory lifecycle.Source: §4.4 · p. 7

5 Experiments

The experiments evaluate TokenWorth under joint resource constraints on LifelongAgentBench, using frozen local and API-based backbones. It achieves the highest task success and lowest output-token use across all six settings, while an ablation examines its adaptation components.

5.1 Experimental Setup

  • The evaluation covers DB, OS, and KG with frozen local and API backbones, shared joint resource budgets, and task-success and output-token metrics.Memory capacity is 64 and at most one memory is retrieved per task; results characterize the specified fixed budgets, not a range of budget levels.Source: §5.1 · pp. 7–8

5.2 Results and Analysis

  • Six of six LifelongAgentBench environment–deployment combinations have TokenWorth as the highest task-success method.The main comparison covers database, operating-system, and knowledge-graph environments under local and API-based deployments.Source: Table 1 · p. 8, §5.2 · p. 8
  • TokenWorth has the lowest output-token use in all six settings, with lower usage than the most generation-efficient competitor while retaining its success lead.Against Standard, OS output-token use falls by 51.5% locally and 58.3% under API deployment.Source: §5.2 · p. 9, Figure 3 · p. 9
  • On local DB, the base scorer reaches 64.2%, fixed-metric RFM falls to 63.0%, and the full AGOP-adapted framework reaches 65.0%.The ablation compares Base Only, Base + RFM, and Full; the authors interpret the pattern as showing that nonlinear residual modeling alone does not necessarily help.Source: §5.2 · p. 9, Figure 3 · p. 9

6 Conclusion

TokenWorth adapts retrieval and bounded memory management using online execution feedback, without updating the backbone. Across three LifelongAgentBench environments and local and API-based deployments, it achieves the highest task success rates while using the fewest output tokens among evaluated methods; feedback only for selected memories leaves their causal contributions unidentified.

6 Conclusion

  • Across three LifelongAgentBench environments and local and API-based backbones, TokenWorth achieves the highest task success rates among evaluated methods while using the fewest output tokens.Additional experiments show competitive performance on coding and embodied-agent tasks.Source: §6 · p. 9
  • TokenWorth learns query-dependent memory worth from online execution feedback and uses it for retrieval and bounded memory management without modifying the backbone.Source: §6 · p. 9
  • Because feedback is observed only for selected memories, their causal contributions remain unidentified.The authors identify budget-efficient credit assignment as future work.Source: §6 · p. 9

A.1 Benchmarks

The appendix describes LifelongAgentBench tasks and additional benchmarks, the comparison methods and shared controls, and the frozen backbones used for local and API-based deployment.

A.1 Benchmarks

  • DB, OS, and KG cover database interaction, multi-step Bash system management, and knowledge-graph tool use under sequential task processing with reuse limited to preceding tasks.Online lifelong variants of BigCodeBench and ALFWorld are also evaluated; experiments focus on fixed resource budgets.Source: §A.1 · p. 13

A.2 Baselines

  • Comparisons include Standard, RAG, and memory- and skill-learning methods, with the same task sequence, backbone, interface, and resource budget within each setting.Baseline-specific auxiliary operations are preserved and their output tokens count toward the budget and reported usage.Source: §A.2 · p. 13

A.3 Backbones and Deployment Settings

  • Local runs use frozen Qwen2.5-14B-Instruct on A800 GPUs, while API-based runs use frozen DeepSeek-V4-Flash; the memory bank starts empty and backbone parameters remain unchanged.Task order is fixed, with no offline pretraining, task replay, or counterfactual re-execution.Source: §A.3 · p. 14

A.4 Evaluation Protocol

The evaluation measures task success and per-task output-token use, counting language-model generation for both task execution and method-specific auxiliary operations.

A.4 Evaluation Protocol

  • Task success rate measures effectiveness, while average output tokens per task measures generation efficiency and includes task-execution and auxiliary-call tokens.Source: §A.4 · p. 14
  • OutputTok excludes input tokens and controller-side computation, so it should not be interpreted as total inference cost.Source: §A.4 · p. 14

B.1 Common Online Implementation

TokenWorth updates worth estimates and memory-bank decisions from task feedback while operating with bounded, environment-adapted resources and memory construction.

B.1 Common Online Implementation

  • Each task runs once using the current bank; its reward updates the worth model and selected-memory statistics, then informs the next bank, with no same-task counterfactual or backbone fine-tuning.Source: §B.1 · p. 14
  • Lightweight hashed text vectors support candidate retrieval and worth scoring, combining similarity, utility, recency, and relevance through a learned linear base scorer.The representation uses token and character 3–5-gram hashing rather than a learned encoder.Source: §B.1 · p. 14
  • A residual learner uses recent feedback, with KRR modeling residual structure and slower AGOP updates adapting feature geometry; the residual coefficient ramps up over the first three KRR fits.Its execution labels reflect downstream outcomes for selected query–memory pairs, not comparisons among memories.Source: §B.1 · p. 15
  • Only externally verified successful tasks yield candidate memories; admission uses novelty or worth criteria, and full banks evict the lowest probe-averaged-worth memory, with a low-utility replacement fallback.The candidate can itself be evicted after admission.Source: §B.1 · pp. 15–16

B.2 Common Hyperparameters

  • Common hyperparameters bound the feedback windows, scorer and residual updates, and shared memory-management thresholds; further bounds and environment-dependent settings are specified separately.Source: §B.2 · p. 16

B.3 Environment-Specific Configuration

  • TokenWorth uses the same worth-learning method across DB, OS, and KG, while adapting resource limits and online-update intervals to each environment.Table A1 presents the environment-specific resource and adaptation configurations.Source: §B.3 · p. 16, Table A1 · p. 16

B.4 Environment-Specific Memory Construction

  • DB memories are compact demonstrations, OS memories retain transferable Bash patterns, and KG memories store successful tool calls and truncated observations rather than backend-executed SPARQL queries.Across environments, up to 12 memories are screened and at most one experience is injected; KG has a 15-round interaction limit and 4096-character memory limit.Source: §B.4 · p. 17

C Generalization Beyond LifelongAgentBench

The appendix evaluates online lifelong variants of BigCodeBench and ALFWorld under fixed task streams and frozen backbones. TokenWorth performs competitively beyond the three main interactive environments, but does not dominate consistently.

C Generalization Beyond LifelongAgentBench

  • BigCodeBench uses a single pass over 1,140 problems, while ALFWorld evaluates 574 episodes across training and validation phases with memory persisting and updating throughout.Both runs use fixed task sequences, update memory after evaluation, and keep the language-model backbone frozen.Source: C Generalization Beyond… · p. 17
  • Methods share per-call output limits, but auxiliary operations have a separate 512-token-per-call limit and total output usage can differ across methods.TokenWorth makes no auxiliary language-model calls.Source: C Generalization Beyond… · pp. 17–18
  • 43.1% on local BigCodeBench and 73.5% on local ALFWorld exceed the best competitors by 0.2 and 0.3 percentage points, while API BigCodeBench’s 50.6% trails RAG’s 50.9%.The additional benchmarks show competitive transfer rather than consistent dominance beyond the three main interactive environments.Source: Table A2 · p. 18, C Generalization Beyond… · pp. 18–19

D Hyperparameter Sensitivity

On the DB stream, TokenWorth is relatively stable under moderate changes to several hyperparameters, while larger changes to adaptation settings reduce success.

D Hyperparameter Sensitivity

  • 65.0% task success is the default DB result; changing RFM influence η between 0.10 and 0.30 shifts performance by at most 1.2 percentage points.The sensitivity evaluation varies one parameter at a time on the same 500-task DB stream using Qwen2.5-14B-Instruct.Source: Table A3 · p. 19, D Hyperparameter Sensitivity · p. 19
  • 65.2% task success follows reducing the base update interval from 50 to 25, a comparable result to the default configuration.AGOP smoothing between 0.10 and 0.50 leads to differences within 0.6 points.Source: D Hyperparameter Sensitivity · p. 19
  • A 2.6-point drop follows reducing η to 0.05, while changing the KRR update interval from 50 to 25 reduces success by 2.8 points.Changing AGOP smoothing to 0.75 causes a 2.8-point drop; changing its update interval from 200 to 100 or 400 decreases success by 2.4 or 3.0 points, respectively.Source: D Hyperparameter Sensitivity · p. 19, Table A3 · p. 19

Related papers

Loading 2610.26453v2…