PaperlayerView exact revision on arXiv

Source-linked AI summary

Coding Agent Memory Post-training: Unlocking the Memory Potential of Pre-trained File Operations for Long-Horizon Tasks via Reinforcement Learning

Lirui Luo, Kelong Mao, Heming Xia, Rongqing Li, Xinwei Yang, Luyu Chen, Kieran Wong, Yudong Guo, Xinrui Wang, Jiayin Zhu, Simiu Gu, Sulong Xu, Cong Fang

arXiv:2609.34422v1cs.LGcs.AIcs.CL

TL;DR

Long-horizon agents can lose task-relevant history when it falls out of active context, and prior memory-RL approaches often depend on environment-specific tools. The paper introduces CAMG and jointly trains CAMG-RL to use files as memory; CAMG-RL-4B leads average success across CAMG’s four environments and both trained models are competitive with larger Qwen3.5 models on two benchmarks.

  • Problem

    Long-horizon interactions can push information needed for later decisions out of active context, while prior memory-RL often relies on environment-specific tools and leaves memory beyond active context an open question.

    Source: §1 · pp. 1–2

  • Method

    CAMG provides four long-horizon environments with shell access and persistent workspaces, while CAMG-RL jointly trains one policy across them using downstream reward and fully asynchronous PPO.

    Source: §1 · p. 2

  • Results

    CAMG-RL-4B achieves the highest average success rate across CAMG’s four test environments, and CAMG-RL-4B and CAMG-RL-9B are competitive with larger Qwen3.5 models on SWE-bench Verified and MLE-bench Lite, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The reported advantage is attributed to fully exploiting and channeling the pre-trained preference for file operations.

    Source: §7 · p. 10

    The study limits memory to one episode and one user, and trains backbones only up to 9B.

    Source: §7 · p. 10

Abstract

from arXiv · show

Language-model agents increasingly tackle long-horizon tasks whose interaction histories exceed the model's active context. Recent work has begun to use reinforcement learning to make memory control part of the policy, often relying on predefined memory tools within domain-specific training environments of relatively short horizons. This setup ties learned memory behavior to environment-specific interfaces that lie outside the base model's pre-training and must be learned from scratch, so even after post-training, agents struggle to use memory in long-horizon tasks. To address these limitations, we introduce Coding Agent Memory Gym (CAMG), a suite of long-horizon agentic-RL environments spanning Shop, Coding, DeepResearch, and AutoResearch. Alongside each environment's native task interface, CAMG provides executable shell access and an episode-persistent workspace, enabling agents to create, revise, search, and reuse files as memory throughout an episode. We also introduce CAMG-RL, which trains a single policy jointly across all four environments with fully asynchronous PPO, learning this file-based memory behavior directly from downstream task reward, and we train CAMG-RL-4B and CAMG-RL-9B from Qwen3.5 models of matching size. On SWE-bench Verified and MLE-bench Lite, CAMG-RL-4B and CAMG-RL-9B are competitive with Qwen3.5-35B-A3B and Qwen3.5-122B-A10B, respectively.

1 Introduction

The paper introduces long-horizon environments and a jointly trained policy that uses ordinary files as memory, then reports strong average performance across CAMG and competitive results on two external benchmarks.

1 Introduction

  • Prior memory-RL work often uses predefined, environment-specific tools, while whether code pre-training can support memory beyond active context remains open.The paper notes that agents trained with such interfaces still struggle with memory on long-horizon tasks.Source: §1 · pp. 1–2
  • CAMG spans Shop, Coding, DeepResearch, and AutoResearch, pairing each native task interface with shell access and an episode-persistent workspace for file-based memory.Agents can create, revise, search, and reuse files throughout an episode.Source: §1 · p. 2
  • CAMG-RL jointly trains one policy across all four environments with downstream task reward and fully asynchronous PPO to use ordinary filesystem operations as memory.The trained CAMG-RL-4B and CAMG-RL-9B models start from matching-size Qwen3.5 models.Source: §1 · pp. 2–3
  • CAMG-RL-4B has the highest average success rate across CAMG’s four test environments, and CAMG-RL-4B and CAMG-RL-9B are competitive with larger Qwen3.5 models on SWE-bench Verified and MLE-bench Lite, respectively.The comparison is with Qwen3.5-35B-A3B on SWE-bench Verified and Qwen3.5-122B-A10B on MLE-bench Lite.Source: §1 · p. 3

2 Related Work

The related work distinguishes learned memory control, context compaction, and agentic-RL environments; CAMG combines filesystem memory with verifiable, cross-context tasks across multiple domains.

2 Related Work

  • CAMG-RL elicits memory through ordinary filesystem actions rather than introducing special memory tools, building on code pre-training.Earlier learned-control approaches use dedicated external-store operations or shorter-lived working and episodic state.Source: §2 · p. 3
  • Context-compaction methods retain state in active context or inside the policy, whereas CAMG-RL stores it in a bounded continuation file at each context replacement.The passage names manager-based compression, bounded internal state, and policy-generated summaries as prior approaches.Source: §2 · p. 3
  • CAMG provides verifiable environments across four task families that require memory across context boundaries, with native task interfaces, shell access, and persistent workspaces.The related-work comparison places CAMG among agent-memory benchmarks and trainable agent environments.Source: §2 · p. 3, Table 1 · p. 3

3 Preliminaries

The preliminaries model each environment as a finite-horizon partially observable decision process and describe policy optimization through expected task return and a vanilla policy gradient.

3 Preliminaries

  • Each environment is modeled as a finite-horizon partially observable Markov decision process with instruction, state, action, observation, transition, and reward components.At each turn, the policy responds to visible interaction history; the environment executes the response and returns an observation.Source: §3 · p. 4
  • A trajectory’s task return sums its rewards over realized turns, discounted by γ.Trajectories have a realized length bounded by the environment’s horizon.Source: §3 · p. 4
  • The policy objective is expected task return, and the section introduces the vanilla policy gradient for its optimization.The trajectory distribution is induced by the policy and environment.Source: §3 · p. 4

4 Coding Agent Memory Gym

CAMG combines four long-horizon task environments with a shared filesystem interface so agents can preserve and reuse information across an episode.

4 Coding Agent Memory Gym

  • CAMG combines verifiable tasks, executable shell access, and an episode-persistent workspace across four environments whose tasks require reuse of earlier information.The environments pair these memory demands with their native task interactions and rewards.Source: §4 · p. 4

4.1 Constructing Four Long-Horizon Environments

  • Shop tests retaining purchase requirements; Coding tests multi-context software repair; DeepResearch tests reuse of retrieved evidence; AutoResearch tests retaining hypotheses and validation evidence.Shop uses six-session tasks; the retained datasets comprise 11,218 Coding tasks, 26,597 DeepResearch tasks, and 931 AutoResearch tasks.Source: §4.1 · p. 4

4.2 A Common Coding-Agent Interface

  • A shared shell_command lets the policy create, inspect, search, modify, and execute files to preserve and revise task-relevant evidence and intermediate state.A command may produce both native-task and filesystem effects, so the action sets can overlap.Source: §4.2 · p. 5

5 CAMG-RL

CAMG-RL treats persistent, editable files as long-horizon memory and learns their use from downstream task rewards. Fully asynchronous PPO lets training proceed across variable-duration episodes without waiting for the slowest rollout.

5.1 Learning to Use Files as Memory

  • CAMG-RL places task interactions and filesystem operations on one trajectory, using GAE and clipped PPO to optimize sampled tokens, including generated file contents.It estimates one advantage per response and clips one ratio per sampled token, combining stable response-level estimation with fine-grained updates.Source: §5.1 · pp. 5–6
  • Persistent, editable files serve as the policy’s long-horizon memory state across the four-environment training objective.Rollouts sample an environment uniformly from Shop, Coding, DeepResearch, and AutoResearch.Source: §5 · p. 5, §5.1 · p. 6

5.2 Fully Asynchronous PPO for Variable-Duration Episodes

  • A persistent episode queue absorbs uneven rollout completion times while one shared actor and critic are updated asynchronously.Rollout workers append completed episodes; the learner assembles them into batches, updates the shared models, and makes the actor available to later episodes.Source: §5.2 · p. 6

6 Experiments

CAMG-RL is evaluated against memory baselines on four environments, with additional tests of transfer, memory components, and saved-memory effects. The results favor filesystem-based memory, while the study’s scope remains limited to one episode and one user, with backbones up to 9B.

6.1 Experimental Setup

  • Trainable methods use Qwen3.5-4B for 200 updates and 12,800 episodes across the four environments; evaluation uses 128 test tasks per environment under matched conditions.Baselines include training-free memory controls and trainable context-compaction and tool-based memory methods.Source: §6.1 · p. 6

6.2 Results

  • 54.4% versus 48.0% average success: CAMG-RL leads CompactionRL, the strongest baseline, on CAMG’s test set.Shop is nearly saturated by both methods; the average gap opens on Coding and AutoResearch.Source: §6.2 · p. 7, Table 2 · p. 7
  • Filesystem memory actions are less surprising to the frozen base than AgeMem’s dedicated tools, while CAMG-RL’s complete memory chains rise from 8.0% to 20.9% of episodes during training.AgeMem attempts its tools in 53 of 261,941 responses and never completes a store–retrieve–use chain.Source: §6.2 · p. 7, Figure 3 · p. 8
  • CAMG-RL-4B and CAMG-RL-9B are competitive with Qwen3.5-35B-A3B and Qwen3.5-122B-A10B, respectively, on SWE-bench Verified and MLE-bench Lite.These external benchmarks cover the Coding and AutoResearch domains.Source: §6.2 · p. 7, Table 3 · p. 8
  • 21.1% average success without memory-action gradients; continuation-only training and evaluation reaches 41.8% versus 54.4%, while blank or mismatched saved files reduce success by 14.1 or 11.5 points.The ablations remove memory-response actor gradients or restrict the policy to the continuation file; the intervention changes saved content at the first context replacement.Source: §6.2 · pp. 8–9, Figure 4 · p. 9
  • A Coding trajectory writes a continuation file before context replacement, reads it afterward, verifies the fix, and submits within the 40-step action budget.The trajectory terminates at step 30; hidden tests pass with return 1.00.Source: §6.2 · p. 9, Figure 5 · p. 9

7 Conclusion

  • The study’s memory is limited to one episode and one user, and its trained backbones reach only 9B.The conclusion identifies cross-episode and cross-user memory, and larger backbones, as future directions.Source: §7 · p. 10

A.2 Shared Agent Interface

  • CAMG preserves each environment’s native task interaction and adds a persistent episode workspace accessed through a shared filesystem tool.At a context boundary, the policy must write a valid continuation file; after replacement, it can read that file from the still-mounted workspace.Source: §A.1 · p. 16, §A.2 · p. 18

A.3 Task Examples

The examples show how CAMG pairs each environment’s native interaction with shared file access, then illustrate a Shop policy carrying information across context boundaries. The trajectory examples include file writes and reads alongside native actions.

A.3 Task Examples

  • Figure 6 pairs one task input from each environment with the interaction available to the policy.The examples cover Shop, Coding, DeepResearch, and AutoResearch.Source: Figure 6 · p. 19, §A.3 · p. 19

A.3.1 Shop

  • The Shop task asks the policy to reuse an earlier color preference in a later shopping session.The six-session example uses native search, click, inspection, and purchase actions alongside shared shell access.Source: §A.3.1 · pp. 19–20

A.3.2 Coding

  • The Coding example pairs an iniconfig issue reproduction with tests for a repository patch.The goal is to submit a patch that passes hidden tests while shell access operates on the repository and policy-authored notes.Source: §A.3.2 · p. 20

A.3.3 DeepResearch

  • The DeepResearch example requires answering a question using evidence retrieved through Search and Visit.The shared files can retain document locations, quotations, candidate facts, conflicts, and open questions.Source: §A.3.3 · p. 20

A.3.4 AutoResearch

  • The AutoResearch example asks the policy to build a tabular model, create submission.csv, and submit it to a private grader.The Seattle house-price task supplies training and test files and specifies RMSLE, lower is better.Source: §A.3.4 · p. 21

A.4.1 Shop

  • Six of six Shop purchases were correct, with episode return 1.00, as the policy used workspace notes and continuation checkpoints across session boundaries.The trajectory summary describes successful file-memory use across all four environments; the Shop trajectory carries confirmed attributes through six sessions.Source: §A.4 · p. 21, §A.4.1 · p. 22

A.4.2 DeepResearch

During a 23-action DeepResearch task, the policy checkpointed accumulated evidence at a mid-research context boundary, resumed research, and answered the question. The answer was accepted despite four out-of-corpus visit penalties.

A.4.2 DeepResearch

  • At the context boundary, the policy saved its objective, five evidence items, and next actions before resuming the search for the square’s name.The monument page identified only “the Square in Tullow,” so the name remained unresolved at the boundary.Source: §A.4.2 · pp. 22–23
  • The resumed search lasted twelve more actions and included four out-of-corpus visits before the policy returned its accepted answer.The task asked for the square containing the Fr. John Murphy statue in Tullow, County Carlow.Source: §A.4.2 · pp. 22–23
  • The answer “Market Square” was accepted, yielding episode return 0.92 after four out-of-corpus visit penalties.The policy reached the answer after writing and rereading a checkpoint at action 9–10, then continuing research through action 23.Source: §A.4.2 · pp. 22–24

A.4.3 Coding

The Coding trajectory repairs a Scrapy response-logging issue, crosses a context boundary before verification, and completes repository tests before submission. The hidden tests pass and the episode return is 1.00.

A.4.3 Coding

  • Before the round-18 context boundary, the policy records the applied fix, persisted test script, and next action in its checkpoint.The policy reads the checkpoint at round 19 and reruns the script at round 20.Source: §A.4.3 · p. 24
  • The policy materializes the issue reproduction as test_spider.py, confirms the scheduler defect, and applies a scripted fix.The fix adds the missing _mqpush method; subsequent repository tests show response logging and the issue resolution.Source: §A.4.3 · pp. 24–25
  • 1.00 episode return; hidden tests pass after the policy repairs the Scrapy issue and verifies the fix.The trajectory terminates at round 30 within Coding’s 40-round budget.Source: §A.4.3 · pp. 24–25

A.4.4 AutoResearch

The AutoResearch trajectory recovers a persisted draft after a context replacement, fixes a holdout-split bug, and submits a valid prediction file. The associated CAMG-RL method trains on native and filesystem actions, assigns later task outcomes to earlier responses, and uses asynchronous PPO.

A.4.4 AutoResearch

  • After the context boundary, the policy found that train_test_split[0] was reused for training and validation, fixed the split, and measured validation_accuracy=0.9241.The policy then created submission.csv and submitted it.Source: §A.4.4 · pp. 25–26
  • 1.14 episode return; the repaired AutoResearch script produced a valid submission accepted by the native grader.The return comprises execution milestone 0.05, validation milestone 0.10, and terminal reward 0.99.Source: §A.4.4 · p. 26

B.1 Policy Trajectory

  • CAMG-RL treats each complete response, including filesystem operations, as one RL step, allowing later task rewards to update earlier memory responses.A context reset changes the next history but adds no sampled tokens or PPO loss term.Source: §B.1 · p. 26

B.2 Credit Assignment

  • Every sampled token in a response receives the same normalized response-level advantage, so longer responses contribute more actor-loss terms.The critic cannot see hidden native state or files the policy has not read.Source: §B.2 · p. 27
  • The no-gradient control omits file-access responses from the actor loss while leaving rewards, returns, critic targets, and critic loss unchanged.Such responses still affect the workspace and consume the same action, token, runtime, and failure budgets.Source: §B.2 · p. 27

B.3 Asynchronous Training

  • Asynchronous training lets rollout and learning proceed independently, with 64 episodes per parameter update and PPO minibatches of 510 responses.Only complete episodes enter the queue; variable episode durations make the environment mixture vary across learner updates.Source: §B.3 · pp. 27–28

B.4 Training Configuration

CAMG-RL jointly trains one actor and critic across four environments using shared settings and native task rewards. Training dynamics are tracked on rollouts, with the final checkpoint evaluated in Table 2.

B.4 Training Configuration

  • CAMG-RL jointly trains on four environments with equal episode allocation and updates all actor and critic parameters.Table 5 gives the shared configuration; Table 6 defines sparse, success-based task rewards.Source: §B.4 · p. 28, Table 5 · p. 29, Table 6 · p. 30
  • At the response limit, the terminal reward goes to the final sampled response, while Shop retains rewards from earlier successful purchases.These rewards are used in action-axis GAE.Source: §B.4 · p. 29
  • Figure 7 tracks training-rollout success over 200 updates using temperature-1 sampling and 64 episodes per update; all four environments improve.Lines are non-overlapping five-update bucket means, and the final bucket corresponds to the checkpoint evaluated in Table 2.Source: §B.4 · p. 30, Figure 7 · p. 31

C.1 CAMG Evaluation

This section specifies CAMG evaluation and reports analyses of memory use, interface priors, and memory restrictions. The evaluation uses matched tasks and conditions across methods, with success rates averaged equally across environments.

C.1 CAMG Evaluation

  • Evaluation uses the same 512 test tasks, decoding, action budgets, runtimes, interfaces, and graders; Average Success is the unweighted mean of four environment rates.The 128 tasks per environment are disjoint by product, repository, question group, or task family, depending on the environment.Source: §C.1 · pp. 31–32
  • Restricting the terminal full-memory policy to a continuation file preserves more of Coding, DeepResearch, and AutoResearch than retraining under that restriction, but Shop falls to 0%.The fully trained policy continues trying to write additional memory files, whereas the restricted-training policy consolidates memory in the continuation file.Source: §C.1 · p. 32
  • Mem0 uses 2,778 high-level memory steps across the 512 test tasks, while Letta Code issues 110 explicit MemFS actions among 13,399 action rows.Mem0’s manager averages 5.43 steps per task; native task actions decrease from 12,535 for the frozen base to 10,603 in the same evaluation.Source: §C.1 · p. 32
  • CAMG-RL’s Average Success interval is [51.6, 57.2], versus [45.4, 50.6] for CompactionRL; their paired 6.4-point difference has interval [3.3, 9.5].The paired difference is significant overall; per-environment advantages are significant on Coding and AutoResearch, while DeepResearch and Shop intervals include zero.Source: §C.1 · p. 32
  • Blanking saved memory lowers average success by 14.1 points, while replacing it with same-environment shuffled content lowers it by 11.5 points.The intervention forks trajectories after the first context replacement while keeping the policy, task state, budgets, and interface fixed.Source: §C.1 · pp. 32–33
  • In updates 181–200, 60.5% of episodes crossed a context replacement; among these, 41.3% stored beyond continuation, 71.6% retrieved earlier memory, and 34.8% of all episodes completed a full chain.Source: §C.1 · p. 33
  • Deleting shell documentation leaves foreign tool names expensive, while the nonce-name control barely changes filesystem-action surprisal.The shell-deletion probe reports a +0.354 nats-per-token store gap; the nonce-name control changes surprisals by at most 0.013.Source: §C.1 · p. 34
  • CAMG-RL attempts memory operations in 18.7–30.2% of responses across quarters and is the only compared policy to acquire complete memory chains.AgeMem makes 53 attempts across 261,941 responses and completes no chain; 49 attempts fail to parse.Source: §C.1 · p. 34

C.2 External Evaluation

The external evaluation tests transfer on two benchmarks using shared tasks and evaluation conditions, then reports precision and contamination controls. Additional Results previews further analyses of interface priors, memory content, baselines, and training runs.

C.2 External Evaluation

  • External transfer is evaluated on SWE-bench Verified and MLE-bench Lite with matched tasks, decoding, action budgets, runtimes, and graders; neither benchmark is used for training.SWE-bench scores issue resolution on 500 tasks, while MLE-bench Lite scores Any Medal rate across 22 competitions.Source: §C.2 · p. 35
  • SWE-bench Verified intervals overlap almost completely for CAMG-RL-4B [12.7, 19.3] and Qwen3.5-35B-A3B [12.5, 19.1]; one MLE-bench medal shifts Any-Medal rate by 4.5 points.The intervals are broad for MLE-bench Lite: one-medal models share [0.1, 22.8], and two-medal models share [1.1, 29.2].Source: §C.2 · p. 35

D Additional Results

  • Additional Results covers interface-prior analyses, a token-level estimation control, memory-content interventions, AgeMem comparisons, failure analyses, run robustness, and training time.The appendix overview also lists the apply_patch variant and wall-clock reporting.Source: D Additional Results · p. 35

D.1 Interface-Prior Name-Token Analysis

The additional analyses examine interface priors, estimation choices, saved-memory effects, baseline comparisons, failure composition, robustness, and related work. Across these studies, the paper contrasts filesystem-based memory with dedicated tools and reports conditions under which results vary.

D.1 Interface-Prior Name-Token Analysis

  • Interface-prior analysis finds the filesystem action rendering favored over dedicated memory tools, while token-level GAE trails the default by 4.3 average points.Deleting shell documentation leaves foreign tool names expensive; the estimation control loses 7.8 points on Coding, 7.7 on DeepResearch, and 3.1 on Shop, while AutoResearch gains 1.6.Source: §D.1 · p. 36, §D.2 · pp. 36–37, Table 9 · p. 37

D.3 Memory-Content Intervention

  • Blanking saved content cuts average success by 14.1 points and shuffling task-mismatched content cuts it by 11.5, with the largest blank drop in Shop and shuffled drop in DeepResearch.Both interventions preserve file paths and filesystem actions while changing memory contents.Source: §D.3 · p. 37, Figure 8 · p. 37

D.4 AgeMem under Its Original Recipe

  • The original AgeMem training recipe improves some environments but does not close the average-success gap; its store–retrieve–use chain rate is 14.1% versus CAMG-RL’s 20.9%.The recipe gains 42.6 points on Shop and 19.6 on DeepResearch, but falls 1.6 and 7.0 points on Coding and AutoResearch relative to common-training AgeMem.Source: §D.4 · p. 38, Table 10 · p. 38

D.5 The DeepResearch Drop of Common-Training AgeMem

  • Common-training AgeMem falls below its frozen base on DeepResearch, accepting 28.1% of answers versus 36.7%; the reported failure mechanism is premature finalization before compaction.The original recipe recovers DeepResearch to 47.7%.Source: §D.5 · p. 38

D.6 The Coding Failures of CAMG-RL

  • Of CAMG-RL’s 94 Coding failures, 58 submit patches that fail hidden tests and 35 exhaust the 40-action budget without submitting.The policy writes and rereads its continuation checkpoint at every context boundary in all 94 episodes.Source: §D.6 · pp. 38–39

D.7 A Single Out-of-Distribution Tool

  • Routing file edits through one apply_patch tool yields 48.6% average success versus 54.4% for CAMG-RL, with 11.7% of responses rejected after training.The variant completes memory chains in 15.6% of episodes, below CAMG-RL’s 20.9%.Source: §D.7 · p. 39, Table 11 · p. 39

D.8 Robustness Across Training Runs

  • Across five runs, CAMG-RL averages 55.3 with standard deviation 2.9, versus 47.1 with 3.5 for CompactionRL; all formal training runs take roughly 120–140 GPU-hours.The single-run averages fall within the reported run intervals, and every formal run finishes within about one day on six GPUs.Source: §D.8 · p. 39, §D.9 · p. 39, Table 12 · p. 40

E Extended Related Work

  • Related-work sections distinguish CAMG-RL from memory benchmarks, context compression and editing, memory-specific credit assignment, and asynchronous agentic-RL systems.The paper characterizes CAMG-RL as learning cross-context file memory from task reward with one asynchronous policy across four task families.Source: E Extended Related Work · pp. 40–41
Loading 2609.34422v1…