PaperlayerView exact revision on arXiv

Source-linked AI summary

RefCon: Iterative Refinement and Contrastive Memory Extraction for Context-Evolving Agent

Ubaidillah Ariq Prathama, Bo Liu, Yeo Boon Hong, Yu-Xuan Huang, Yangkai Ding, Tao Yu

arXiv:2609.39143v1cs.AIcs.LGcs.MA

TL;DR

Long-horizon agent trajectories contain reusable but noisy experience, while retraining to absorb it is expensive and prior scaling approaches are often studied separately. RefCon combines sequential self-refinement with parallel self-contrast to extract memories without gold labels. Across benchmarks and agent frameworks, it reports consistent gains, including improvements over no-scaling baselines.

  • Problem

    Agent trajectories can contain reusable experience but are often discarded, while retraining is expensive and sequential and parallel scaling are typically studied separately.

    Source: §1 · pp. 1–2

  • Method

    RefCon generates trajectories through sequential self-refinement and jointly analyzes them with parallel self-contrast to distill reusable insights without ground-truth labels.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across AppWorld and BFCL-V3, RefCon is strongest or near-strongest with favorable accuracy-token trade-offs; it improves performance by 21.6% on ACE over no-scaling.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    RefCon remains effective across model scales and generalizes to SWE-bench, where it surpasses some ground-truth baselines.

    Source: §5 · p. 8

    Generalization to model architectures or tuning styles beyond the three evaluated scales is not fully guaranteed, and scaling was tested only up to five trajectories.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Long-horizon agent interactions generate useful but noisy experience, and retraining models to absorb it is expensive. Context-evolving agents therefore need memory extraction methods that improve with more test-time compute without relying on gold labels. We propose RefCon, which combines sequential self-refinement with parallel self-contrast to extract higher-quality memories without gold labels. Evaluated on AppWorld and BFCL-V3 across multiple context-evolving agent frameworks, RefCon delivers strong and consistent gains, including relative improvements of 21.6% on ACE and 16.6% on ReMe over no-scaling baselines, while a diversity-focused variant (DivCon) achieves a 35.5% gain on ReasoningBank. RefCon consistently outperforms existing baselines without ground-truth labels, and generalizes across model scales and to software engineering tasks, where it surpasses even ground-truth baselines. We further analyze the accuracy-token trade-off and scaling behavior, showing RefCon maintains favorable efficiency and continues to improve as more trajectories are used, unlike diversity-only scaling which saturates earlier.

1 Introduction

Long-horizon agent interactions yield reusable experience, but trajectories are often discarded and retraining is expensive. RefCon combines sequential refinement with parallel contrast to improve memory extraction, with gains reported across agent frameworks.

1 Introduction

  • Context-evolving agents write, refine, and retrieve memories during inference, while memory extraction distills trajectories and retraining remains computationally expensive.Agents also retrieve relevant experiences and manage memory stores by filtering redundancy and irrelevant content.Source: §1 · p. 1
  • Prior work typically studies sequential and parallel scaling separately, although the former enhances trajectory quality and the latter improves diversity.Evaluating extraction requires recurring tasks and ambiguous trajectories so good and poor methods can be distinguished.Source: §1 · pp. 1–2
  • RefCon generates trajectories through self-refinement, then uses self-contrast across them to identify improvements and distill reusable insights.DivCon instead promotes diversity by exploring alternative strategies rather than refining previous trajectories.Source: §1 · p. 2
  • RefCon unifies sequential and parallel test-time scaling to improve memory extraction quality, and is proposed as a plug-in for agentic systems.The framework is intended to support context-evolving methods and future research.Source: §1 · p. 2
  • 21.6% on ACE and 16.6% on ReMe over no-scaling baselines; DivCon gains 35.5% on ReasoningBank, while RefCon outperforms existing baselines without ground-truth labels.Source: §1 · p. 2

2 Related Works

Related work covers memory extraction, retrieval, and management, with systems using different strategies to distill experience, find relevant memories, and control memory growth.

Memory Extraction

  • Memory extraction ranges from synthesized memory updates and cautionary insights to high-level heuristics, task-specific workflows, and reusable code snippets.ACE separates reflection from curation, while ReasoningBank incorporates insights from failed trajectories.Source: §2 · p. 2
  • ReasoningBank introduced memory-aware test-time scaling using self-contrast for parallel scaling and self-refine for sequential scaling, which follow-up work extended.Source: §2 · p. 3

Memory Retrieval

  • Memory retrieval has progressed from inserting all memories or basic similarity search to reranking, agent-led research, graph traversal, and learned tool calls.ReMe uses a retrieve–rerank–rewrite pipeline.Source: §2 · p. 3

Memory Management

  • Memory management systems use dedicated agents for insert, update, and delete operations or prune memories by retrieval utility, but explicit management remains underexplored.Source: §2 · p. 3

3 Methodology

The method treats agent learning as online, task-level memory updates without access to future queries or ground-truth labels. RefCon refines and contrasts trajectories to extract memories, then retrieves and prunes memory for later tasks.

3.1 Problem Formulation

  • The agent processes sequential task queries without future queries or ground-truth labels, updating task-level memory from prior trajectories to retain reusable insights.Its policy conditions on the current memory and available action/tool space.Source: §3.1 · p. 3

3.2 RefCon for Memory Extraction

  • RefCon generates and iteratively refines trajectories, then contrasts them to identify successful reasoning patterns and distill candidate memories.The resulting attempts progress from initial exploration toward refined, self-corrected strategies.Source: §3.2 · pp. 3–4
  • Candidate memories are deduplicated against existing and newly proposed entries, while each retained memory stores when-to-use guidance and insight content.Entries may also include frequency and utility metadata.Source: §3.2 · p. 4

3.3 Memory Retrieval and Management

  • For each query, retrieval, reranking, and rewriting produce concise, query-aligned prior experience; utility-based deletion prunes low-average-utility memories after sufficient retrievals.Removal requires retrieval at least α times and average utility below β.Source: §3.3 · pp. 4–5

4 Experiments

Experiments compare context-evolving agents on AppWorld and BFCL-V3, then examine combined scaling, coding-task generalization, token costs, and scaling behavior. RefCon performs strongly without ground-truth labels, while DivCon leads on ReasoningBank in one comparison.

4.1 Setup

  • Experiments use AppWorld and BFCL-V3 with GLM-4.6, comparing ReAct and three context-evolving agents; the label-free MaTTS comparison disables ground-truth use.AppWorld emphasizes recurring multi-step app tasks; BFCL-V3 uses 50 multi-turn travel tasks, and additional experiments use Gemma 4 31B and Qwen3.5 9B.Source: §4.1 · p. 5

4.2 Main Comparison

  • Without ground-truth labels, RefCon is best overall, improving over ReAct by 16.51% on AppWorld Avg@3 (TGC), 50.06% (SGC), and 32.36% on BFCL-V3 Pass@3.It surpasses some ground-truth baselines and nearly matches the top AppWorld scores; self-refinement can degrade correct BFCL-V3 trajectories, while ReMe’s parallel contrastive signal is frequently absent.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 RefCon’s Combined Scaling Advantage

  • On AppWorld, RefCon improves over no scaling by 21.6% on ACE and 16.6% on ReMe, while DivCon leads on ReasoningBank with a 35.5% gain.RefCon benefits more from progressively improved trajectories, whereas DivCon benefits from diversity; the paper describes DivCon as less stable with weaker scaling behavior.Source: §4.3 · p. 6, Table 2 · p. 7

4.4 Generalization to Coding Tasks

  • On SWE-bench Verified (Mini), RefCon outperforms all baselines, including ACE and ReMe (Sequential) methods that use ground-truth.Its self-contrast reasoning across multiple trajectories enables it to outperform ACE even when ACE receives ground-truth code patches.Source: Table 3 · p. 7, §4.4 · p. 7

4.5 RefCon Accuracy-Token Tradeoff

  • RefCon consistently achieves the highest accuracy across methods—72.53% on ACE, 71.93% on ReasoningBank, and 74.30% on ReMe—with 0.17M–1.18M (≤3%) prompt-token overhead over sequential scaling and gains up to +8.2%.Trajectory generation via self-refinement is no more costly than sequential scaling, while one-time extraction consolidates all three refined trajectories.Source: §4.5 · pp. 7–8

4.6 Scaling Law

  • RefCon and DivCon both reach about 71.5% at k = 2 from a shared 63.7% baseline at k = 1, but RefCon reaches 76.0% at k = 5 while DivCon plateaus around 71.0%.The authors conclude that solution quality is a more reliable driver than diversity as trajectories scale.Source: Figure 5 · p. 8, §4.6 · p. 8

5 Conclusion

RefCon combines sequential self-refinement with parallel self-contrast to extract higher-quality memories without gold labels. It performs strongly across benchmarks and agent frameworks, with evidence of efficiency and transfer to software engineering tasks.

5 Conclusion

  • RefCon combines sequential self-refinement with parallel self-contrast to extract higher-quality memories for context-evolving agents without gold labels.The framework is presented as a memory-aware test-time scaling approach.Source: §5 · p. 8
  • RefCon’s label-free extraction surpasses some ground-truth baselines and achieves strong or near-strongest performance across AppWorld and BFCL-V3.The conclusion also reports favorable accuracy-token trade-offs.Source: §5 · p. 8
  • Applying RefCon to SWE-bench supports its generalizability to intricate, real-world software engineering environments.The conclusion frames memory extraction at test time as a practical, compute-efficient path to continual agent improvement.Source: §5 · p. 8
  • DivCon indicates that explicit diversity can help, while also revealing stability and scaling limits.The conclusion motivates further work on more robust diversity generation and adaptive scaling strategies.Source: §5 · p. 8

Limitations

The study identifies boundaries in model-family generalizability and in the compute range covered by its scaling analysis.

Limitations

  • Results across GLM-4.6, Gemma 4 31B, and Qwen3.5 9B do not guarantee generalizability to models with different architectures or tuning styles.The authors plan to evaluate more diverse model families.Source: Limitations · p. 9
  • The scaling analysis covers at most five trajectories, leaving behavior at larger compute budgets uncharacterized.The study cannot yet establish whether performance continues improving, plateaus, or degrades beyond k ≤5.Source: Limitations · p. 9

Ethical Considerations

The work reports no specific ethical concerns and describes its evaluation setting and a possible performance risk from generated memories.

Ethical Considerations

  • The experiments use publicly available benchmarks and open-source models, without human subjects, sensitive data, or privacy-related information.The paper reports no potential conflicts of interest.Source: Ethical Considerations · p. 9
  • Generated memories may occasionally degrade model performance, though the paper describes this as infrequent and mitigable through extraction or retrieval thresholds.Source: Ethical Considerations · p. 9
  • The authors state that the work raises no specific ethical concerns.Source: Ethical Considerations · p. 9

A RefCon and DivCon Detailed Results

At smaller model scales, RefCon remains competitive without ground-truth labels, though gains are less pronounced than with the larger model. On recurring AppWorld tasks, RefCon and other multi-trajectory approaches show strong later-task improvement.

A RefCon and DivCon Detailed Results

  • Without ground-truth labels, RefCon outperformed other context-evolving agents at Gemma 4 31B and Qwen3.5 9B, even surpassing ACE with ground-truth on AppWorld.Source: A RefCon and DivCon Detailed Results · p. 11
  • At smaller scales, RefCon’s gains were less pronounced than with GLM-4.6; Qwen3.5 9B performance declined substantially on AppWorld but only marginally on BFCL-V3.Some context-evolving methods degraded on AppWorld with Qwen3.5 9B, while Gemma 4 31B performed similarly to GLM-4.6 and showed more benefit from memory reuse.Source: A RefCon and DivCon Detailed Results · p. 11
  • Reported RefCon Avg@3 uses its final run after the second refinement, whereas DivCon Avg@3 uses its first run.The authors report these runs because later RefCon runs can improve accuracy, while DivCon’s diversity emphasis does not always yield better scores.Source: A RefCon and DivCon Detailed Results · p. 11
  • All MaTTS methods improved on subsequent related AppWorld tasks, with RefCon, DivCon, and parallel scaling showing the steepest first-to-second-task gains.Sequential scaling began with high first-task performance but did not sustain consistent later improvements, likely because it extracts memory from a single trajectory.Source: A RefCon and DivCon Detailed Results · p. 11

B Memory Examples

The examples place RefCon across ACE, ReasoningBank, and ReMe, whose memory pipelines, scaling strategies, and schemas differ.

B Memory Examples

  • ACE uses a reflector–curator pipeline, ReasoningBank uses MaTTS with embedding-based retrieval, and ReMe adds retrieve–rerank–rewrite filtering.The experiments use ReAct as the base framework.Source: B Memory Examples · p. 12
  • ACE has no default test-time scaling; ReasoningBank and ReMe use different parallel and sequential strategies, with ReMe extracting best, worst, and comparison memories in parallel and failed-trajectory caution memories sequentially.ReasoningBank's MaTTS supports both parallel and sequential scaling.Source: B Memory Examples · p. 12
  • ACE stores section–content pairs, ReasoningBank uses title, description, and content, and ReMe uses when_to_use and content before rewriting retrieved memories as a step-by-step guide.The examples use the same Spotify query and RefCon scaling across frameworks.Source: B Memory Examples · pp. 12–13
  • Tables 6 and 7 report RefCon and DivCon accuracy across runs, respectively, for each method.Source: Table 6 · p. 13, Table 7 · p. 13

C Example of Successful RefCon Improvement

On an AppWorld messaging task, RefCon retrieves memories that support API exploration and case checking, while sequential scaling retrieves guidance that leads to failure.

C Example of Successful RefCon Improvement

  • Figure 6 plots Avg@3 improvement across similar tasks within an AppWorld scenario, sorted by chronological run order.Source: Figure 6 · p. 14
  • RefCon correctly identifies contacts_without_venmo after inspecting the data structure, whereas sequential scaling brute-forces API fields and sends messages to all_contact.The retrieved memories differ in whether they encourage API exploration and case checking or skipping verification.Source: C Example of Successful RefCon… · pp. 13–14
  • A plausible call to show_song_privates silently assigns zero play counts to 18 songs, producing an arbitrary ranking despite valid calls and well-formed output.Both trajectories first collect 79 unique song IDs and filter to 18 R&B candidates; show_song exposes play_count, while show_song_privates does not.Source: C Example of Successful RefCon… · p. 14

D Hyperparameter Details

The appendix specifies trajectory-scaling and retrieval settings, then describes prompt adaptations and examples for RefCon and DivCon.

D Hyperparameter Details

  • The usual scaling factor is 3, except 2 for DivCon; parallel rollouts use temperatures 0.7, 0.85, and 1.0, and extraction is capped at five memories per step.The default temperature otherwise is 0.7.Source: D Hyperparameter Details · p. 14
  • ReasoningBank retrieves up to five memories with top-1 query retrieval, ACE caps its playbook at 50 memories, and ReMe and RefCon rerank top-10 retrievals to choose five.ReMe and RefCon use deduplication threshold ϵ = 0.5 and pruning thresholds α = 5n for usage and β = 0.5 for utility.Source: D Hyperparameter Details · p. 14

E Prompt Details

  • RefCon modifies trajectory counts while retaining prompt instructions, and its extraction combines ReasoningBank self-contrast reasoning with ReMe’s memory format.The appendix provides trajectory-generation prompts for AppWorld, BFCL-V3, and SWE-bench Verified (Mini).Source: E Prompt Details · p. 14, Table 14 · p. 19, Table 15 · p. 19
  • The simpler AppWorld refinement prompt was ineffective on BFCL-V3, so the authors added critique and ideation prompts; self-refinement sometimes degraded already-correct trajectories.Source: E Prompt Details · p. 14
  • Prompt-detail examples include RefCon-scaled memories from ACE, ReasoningBank, and ReMe, plus rules for authentication, pagination, song-ID extraction, and unique counting.Source: Table 8 · p. 15, E Prompt Details · p. 15, Table 10 · p. 16
  • Tables 11–13 cover retrieved-memory comparisons, trajectory comparisons, and step-by-step comparisons of successful and failed trajectories.Source: Table 11 · p. 16, Table 12 · p. 17, Table 13 · p. 18
Loading 2609.39143v1…