PaperlayerView exact revision on arXiv

Source-linked AI summary

ReFold: Training-Free Reversible Inter-Turn Context Folding for Long-Horizon Agents

Yupeng Su, Jiayi Tian, Zheng Zhang, Souvik Kundu

arXiv:2610.07863v1cs.CLcs.AIcs.LG

TL;DR

Long-horizon agents resend their append-only histories at every step, increasing context and inference costs, while existing management approaches can add overhead, disrupt cache reuse, or lose content. ReFold compresses the rendered context through reversible deduplication and folding, and evaluations show lower token use, cost, and per-session KV-cache memory without detectable task-success reduction at the full window.

  • Problem

    Long-horizon agents resend an append-only interaction history at every step, causing context and inference costs to grow with each successive step.

    Source: §1 · p. 2

  • Method

    ReFold is a training-free rendering layer that deduplicates repeated content and folds turns the agent reports finished while preserving the underlying interaction history.

    Source: §1 · p. 2

  • Results

    Across five benchmarks and two models, ReFold reduces token usage by 18–60%, estimated cost by up to 45%, and peak per-session KV-cache memory by 30–58%, with no detectable task-success reduction at the full window with w=8.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Under constrained context budgets and concurrent serving, ReFold’s savings reduce compaction frequency and scheduling delay, improving execution efficiency.

    Source: §6 · p. 9

Abstract

from arXiv · show

Long-horizon LLM agents act on an append-only interaction history that is re-sent to the model at every step, so the context and its cost grow with steps until the sessions exceed the context window. Existing methods manage the context through context requirement prediction, relying on additional model calls, heuristic rules, or trained policies. However, these predictive approaches introduce runtime overhead, invalidate prefix caches, and permanently discard content with no guarantee of recovery. To overcome these limitations, we introduce ReFold: a training-free rendering layer that preserves the underlying interaction history while compressing only the model's rendered context. It removes two kinds of inter-turn redundancy without an auxiliary predictor: content an earlier turn already displayed, replaced by a stub, and turns the agent itself reports finished, folded into a one-line note. Both operators use chunked rendering, rewriting the cached prefix once every few steps rather than at every step. Every removal is strictly reversible, a wrong removal costs one restore from the history rather than permanent content loss. Because it operates at the rendering layer, ReFold is plug-and-play across standard ReAct-style harnesses. Evaluations across five long-horizon benchmarks and two frontier LLMs demonstrate that ReFold reduces token consumption by up to 2.5x and halves the KV-cache memory per session without degrading task success rates. Under capped context budgets, it avoids up to 92% of forced compactions. Under concurrent serving workloads, it reduces request queuing delays by up to 100%, accelerating inference by up to 1.7x, while cutting inference costs by up to 3.4x.

1 INTRODUCTION

Long-horizon agents resend an append-only interaction history at every step, so context and inference costs grow as trajectories continue. ReFold compresses only the rendered context, using repeated-content deduplication and agent-reported completion folding while preserving the underlying history.

1 INTRODUCTION

  • Long-horizon agents resend prior tool calls and environmental observations at every step, causing context and inference costs to grow and accuracy to degrade before the window limit.An analysis of GitHub Copilot found that 7.8% of sessions triggered compaction but accounted for 44.2% of total token usage.Source: §1 · p. 2
  • Existing predictive context-management methods can add runtime overhead, break prefix-cache reuse, or permanently discard context without guaranteed recovery.The approaches include positional eviction, content pruning, external storage, and summarization.Source: §1 · p. 2
  • ReFold is a training-free rendering layer that preserves the append-only interaction history while compressing only the context shown to the model.It replaces previously displayed repeated content with stubs and folds turns the agent reports finished into a one-line note.Source: §1 · p. 2
  • Across five long-horizon benchmarks and two frontier LLMs, ReFold reduces token consumption and halves per-session KV-cache footprint without degrading task success rates.It also achieves lower cost and faster completion under capped budgets or concurrent load.Source: §1 · p. 2

2 MOTIVATIONAL STUDY

A controlled comparison of six strategies on 50 SWE-bench Verified tasks examines context footprints, cost, duration, cache reuse, and evicted-token recoverability. It finds that pruning tool outputs alone cannot bound context, while context-management operations can add overhead and sacrifice cache reuse or recoverability.

2 MOTIVATIONAL STUDY

  • The comparison evaluates six context-management strategies on the same 50 SWE-bench Verified tasks under a 16k context budget.The strategies include mask, prune, recall, compact, clip, and the unpruned base.Source: §2 · p. 2
  • Pruning tool outputs alone cannot guarantee a fixed context budget; enforcing one requires controlling agent commands and reasoning traces as well.These account for context beyond tool outputs, including the fixed system prompt and task description.Source: §2 · pp. 2–3
  • 0.89–0.98 GiB peak KV-cache footprints and 38–86% context-budget exceed rates occur under the five non-compacting strategies; compact reaches approximately 0.69 GiB and 0% exceedance.Mask and recall reduce tool outputs’ share of the peak footprint to approximately 24–29%, but leave agent messages and other context accumulating.Source: §2 · p. 3, Figure 2 · p. 3
  • Compact adds roughly 152 seconds of execution time and approximately 1.6 cents per task through summarization calls; prune adds roughly 5 seconds through scorer calls.Mask also incurs 5.1 cents per task from lost cache, accounting for 31% of its total cost.Source: §2 · p. 3, Figure 2 · p. 3
  • 23% cache-hit rate under mask compares with 84% under base, while clip, mask, prune, and compact leave 17–48% of all observation tokens unrecoverable.The figure tracks cache reuse and the fate of evicted observation tokens; recall can return everything it evicts.Source: Figure 2 · p. 3, §2 · p. 4

3 METHOD: REFOLD

ReFold compresses the context rendered to the model while preserving the append-only interaction history. It combines deduplication and folding with chunked rendering, reversibility, and a rendering-layer integration hook.

3.1 SETUP

  • Each turn couples an agent action, including reasoning and tool calls, with the resulting environment observation, and resending turns makes cumulative context volume grow quadratically.The rendering function operates on this append-only history rather than replacing it.Source: §3.1 · p. 4

3 METHOD: REFOLD

  • ReFold maps the interaction history to a compressed model context without modifying the underlying history.The rendering layer preserves the agent’s tools, environment, and interaction protocol.Source: §3.1 · p. 4, §3 · p. 4, §3.4 · p. 6

3.2 CONTEXT RENDERING

  • Deduplication replaces repeated observation spans with reference stubs, while folding replaces turns the agent reports finished with short notes.Stubs identify the omitted region and retained source turn; folding uses PROGRESS and STALE metadata generated in the regular response.Source: §3.2 · p. 5

3.3 CHUNKED RENDERING AND REVERSIBILITY

  • A shared boundary advances every k steps: newer turns remain full, while earlier turns are folded if marked finished and otherwise deduplicated.Between boundary advances the rendered context remains append-only, preserving reuse of the rendered-prefix KV cache; removed turns can be restored from history.Source: §3.3 · pp. 5–6

3.4 INTEGRATION WITH AGENTIC WORKFLOWS

  • ReFold integrates through a rendering hook and prompt instructions that elicit PROGRESS and STALE, leaving existing ReAct-style tools, environment, and interaction protocol unchanged.The OBSERVE step records turns and signals; RENDER constructs the compressed context before each model call.Source: §3.4 · p. 6

4 EXPERIMENTS

ReFold is evaluated across five benchmarks and two models, then tested under context-budget and serving constraints and ablated to examine its overhead, chunk size, chunking, and reversibility.

4.1 SETUP

  • The evaluation uses two models, five benchmarks with 50 tasks each, a shared harness, and measures task performance, resource use, and execution efficiency.At the full window, base and compact produce identical runs because compact's threshold is never reached.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • Token use falls 18–60%, estimated cost up to 45%, and per-session KV-cache memory 30–58%, with no detectable task-success reduction at the full window and w=8.Across the two models, token reductions are 18–53% and 42–60%, while KV-cache reductions are 39–58% and 30–55%, respectively.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 PERFORMANCE UNDER RESOURCE CONSTRAINTS

  • At 24k–64k budgets, mean cost drops 24–36%, median duration 12–17%, and compactions by up to 92%; at 16k, compactions and duration fall 33% and 14%.Resolved-task counts differ by at most three out of 50 across tested budgets.Source: §4.3 · p. 7, Table 2 · p. 8
  • At w=16–48, token use falls 53–70%, mean cost 46–71%, and median duration 23–41%; at w=48, scheduling delay drops from 8.96 to 0.02 s.With a 16k budget and w=48, token use falls 24%, compactions 37%, cost 17%, duration 28%, and scheduling delay from 0.87 to 0.02 s; both conditions solve 36 of 50 tasks.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 ABLATION STUDIES

  • ReFold adds 0.4 ms of mean rendering overhead per step, versus 3.3 s for compact, 0.15 s for prune, and 8.5 ms for recall.It renders without auxiliary model calls.Source: §4.4 · p. 8
  • Peak KV-cache memory and median duration are lowest at chunk sizes k=3–5, and k=3 is selected to limit prefix-cache invalidation while retaining compression.Resolved-task counts vary little across tested chunk sizes.Source: §4.4 · p. 8, Figure 4 · p. 9
  • Chunked rendering rewrites the cached prefix every k=3 steps and retains an 87% hit rate, versus 51% for per-step rendering and 84% under base.Rewrites are confined to boundary advances, with the context append-only between them.Source: §4.4 · p. 8, Figure 4 · p. 9
  • With reversibility enabled, ReFold removes 41% of observation tokens and every removed token can be restored; disabling it makes about a third unrecoverable.With reversibility off, ReFold removes 53% of observation tokens, the same unrecoverable share that compact loses.Source: §4.4 · p. 8, Figure 4 · p. 9

5 RELATED WORK

Related work addresses long-horizon agent context through observation pruning, summarization, external memory, and prompt or KV-cache compression; ReFold combines reversible removal with chunked rendering.

Context management for LLM agents

External memory and token compression

  • ReFold compresses observations and completed agent turns without auxiliary summarization calls or policy training, and makes removals reversible while chunking preserves prefix-cache locality.The passage contrasts these properties with lossy compaction and rigid external stores.Source: §5 · p. 9
  • External-memory approaches store or retrieve history, while prompt- and KV-cache compression methods reduce input tokens or cache contents.Examples include MemGPT [Packer et al., 2023], LLMLingua [Jiang et al., 2023], and SkipKV [Tian et al., 2026].Source: §5 · p. 9

6 CONCLUSION

ReFold preserves interaction history while reducing the rendered context, with reported resource savings and no detectable task-success reduction at the full window. The paper also describes its implementation and benchmark evaluation setup.

6 CONCLUSION

  • Across five benchmarks and two models, ReFold reduces token use by 18–60%, estimated cost by up to 45%, and peak per-session KV-cache memory by 30–58%, with no detectable task-success reduction at the full window and w=8.Under constrained budgets and concurrent serving, the savings reduce compaction frequency and scheduling delay.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

  • The mini-swe-agent plugin requires a rendering hook and prompt change, while leaving the agent’s tools, environment, and interaction protocol unchanged.The paper says it will release the code and scripts.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 SETUP

  • Runs share a two-GPU serving setup and the same harness, tools, and execution environment.The shared settings are listed in Table 3.Source: §A.1 · p. 16, Table 3 · p. 16

A.2 BENCHMARKS

  • Five benchmarks each contribute 50 stratified tasks, scored by their official evaluators in official containers.The set covers three SWE-bench variants and two Terminal-Bench releases.Source: §A.2 · p. 16

A.3 METRICS

This section defines how benchmark coverage and resource metrics are reported, and qualifies task-resolution comparisons with observed run-to-run variability.

A.3 METRICS

  • Tokens and cost are task means; steps, duration, and KV-cache are medians, with auxiliary model calls included in token and cost totals.The metric definitions also specify token pricing and the components of the KV-cache calculation.Source: §A.3 · p. 16
  • The five benchmarks contribute 50 tasks each; SWE-bench Pro covers all public-set repositories and Multi-SWE-bench spans seven languages.SWE-bench fixes require issue-reproducing tests to pass while previously passing tests remain passing; Terminal-Bench scores the resulting container state.Source: Table 4 · p. 17
  • Paired-row brackets report change from the comparison arm, with colors marking better, worse, or zero/within-noise outcomes.Green means better, red worse, and grey zero or within run-to-run noise.Source: §A.3 · p. 17

A.4 RUN-TO-RUN VARIABILITY

  • Up to four resolved tasks can separate two arms within run-to-run noise; repeated runs can also flip 2–7 tasks in each direction.Six of seven repeated-configuration pairs changed resolve count by 1–3 of 50, while the context-near-limit mask configuration changed by 8.Source: §A.4 · p. 18, Table 6 · p. 18

B.1 THE SIX STRATEGIES

The section defines six context-management strategies, their measurement setup, and ReFold’s algorithm and prompt protocol.

B.1 THE SIX STRATEGIES

  • The six strategies are compared in one harness on the same 50 SWE-bench Verified tasks at a 16k context budget, with each strategy’s own model calls counted.Base retains all history; clip truncates older observations, and mask replaces older observations with placeholders while leaving agent reasoning and commands intact.Source: §B.1 · pp. 18–19, Table 7 · p. 18
  • Prune filters selected file-read and search outputs; recall moves older observations into a retrievable store, while compact replaces older history with a model-generated summary.Recall and compact trigger after estimated prompt use passes 75%; compact summarizes the oldest 60% not yet summarized.Source: §B.1 · p. 19

B.2 HYPERPARAMETER SETTINGS

  • Clip and mask are tested at three knob settings; the section uses the settings that resolve most, while prune, recall, and compact use their specified settings.The selected clip cap is 1,000 characters and the selected mask window is N=10.Source: §B.2 · p. 19, Table 8 · p. 19

B.3 HOW FIGURE 2 IS MEASURED

  • Figure 2 measures cache growth and composition, strategy cost and time, prefix-cache reuse over 30 calls, and the fate of removed observation tokens.Its cache panels isolate the portion that grows with the prompt; removed tokens are classified as recoverable, reproducible, or irrecoverable.Source: §B.3 · pp. 19–20

C.1 ALGORITHM

  • ReFold’s OBSERVE procedure appends each turn and extracts progress and stale-step markings; RENDER constructs the model context using its computed boundary and selected steps.The rendering branches retain full turns, render folded notes, or append deduplication stubs.Source: §C.1 · p. 20

C.2 THE PROMPT LAYER

  • The prompt layer instructs the agent to emit PROGRESS and STALE metadata, explains how to restore omitted output, and labels rendered tool outputs by step.The response-format item is replaced with the two-line protocol, while the rest of the template remains unchanged.Source: §C.2 · pp. 20–21

C.3 WORKED EXAMPLE

The examples show folding and deduplication in operation, including recovery after an incorrect stale marking; the later results report a proprietary-model comparison and per-step and per-task patterns.

C.3 WORKED EXAMPLE

  • Folding steps 1–6 condenses 13,584 characters into a 1,219-character note, shrinking the prompt by over 2,200 characters despite carrying one more turn.The agent marks the steps stale after identifying the root cause, while keeping the read that pinned it down.Source: §C.3 · p. 22
  • A repeated 25-line traceback becomes a stub pointing to its earlier copy, reducing the rendered output from 1,044 characters to 130.The earlier step still carries the full traceback.Source: §C.3 · p. 23
  • When the agent marks a needed step stale, re-running its unchanged command restores the output; one re-run undoes the incorrect removal.In the example, the agent later patches the relevant line and solves the task.Source: §C.3 · p. 23

D.1 A PROPRIETARY MODEL

  • ReFold cuts tokens by 24%, steps by 17%, cost by 11%, and duration by 19% with Claude Opus 4.8 on 21 SWE-bench Verified tasks.Resolve remains within run-to-run noise; the prompt cache serves 94% of base prompt tokens and 92% of ReFold prompt tokens.Source: Table 9 · p. 23, §D.1 · p. 23

D.2 PER-STEP GROWTH

  • On SWE-bench Verified, the prompt-growing KV cache is 15% smaller at step 10, 35% at step 20, and 46% at step 30 under ReFold.The gap widens with trajectory length, and every benchmark shows the same direction.Source: §D.2 · p. 24, Figure 5 · p. 24

D.3 PER-TASK DISTRIBUTION

  • ReFold sends fewer tokens than base on 66 to 90% of tasks in every benchmark-model cell.The per-task distribution shifts toward fewer tokens rather than reducing only a few long tasks.Source: Figure 6 · p. 24, §D.3 · p. 24

D.4 PUBLISHED METHODS ON THE SAME HARNESS

On SWE-bench Verified, the published-method comparison finds that ReFold reduces tokens and cost more than the alternatives and is the only method that halves session memory.

D.4 PUBLISHED METHODS ON THE SAME HARNESS

  • ReFold cuts tokens by 48% and cost by 34%, several times more than any other method, and only ReFold halves per-session memory.Methods that drop lines or tokens within an observation cannot remove the whole repeated observations that set the peak.Source: Table 10 · p. 25, §D.4 · p. 25
  • AgentDiet’s cost rises with rewrite frequency; at θ=150, its auxiliary generation is three times its main loop.Its rewriting calls count toward tokens and cost.Source: §D.4 · p. 25
  • Resolve stays within three tasks of base for every compared method, inside the reported run-to-run noise.Source: §D.4 · p. 25

E TWO OPERATORS

Folding accounts for most of ReFold’s reductions, and combining folding with deduplication leaves the smallest KV cache. Its rendering hook and prompt instructions also allow it to run beneath other mechanisms.

E TWO OPERATORS

  • With both operators on, ReFold cuts per-session KV cache by 52%; folding alone cuts it by 49%, while deduplication alone cuts it by 16%.Token reductions are 50% with folding alone and 33% with deduplication alone; completion tokens are at most 3% in every cell, so differences are on the prompt side.Source: E TWO OPERATORS · p. 25, Table 11 · p. 25

F COMPOSITION WITH OTHER MECHANISMS

  • ReFold needs only a rendering hook and prompt instructions, allowing it to run beneath mechanisms that act elsewhere in the loop, including compaction.The section also tests composition with mechanisms in the serving system, model choice, and prompt.Source: F COMPOSITION WITH OTHER MECHANISMS · p. 25

F.1 ROUTING ACROSS REPLICAS

Across replica routing, model alternation, and retrieval memory, the passages report ReFold’s savings and task-resolution results under each setting. The outcomes vary by mechanism and load.

F.1 ROUTING ACROSS REPLICAS

  • At w=32 under round-robin routing, ReFold resolves 40 tasks against base’s 26 in 41.8 against 75.5 minutes.Under sticky routing, the arms resolve within four tasks of each other at both loads; ReFold cuts tokens by 55% and 42% and duration by 24% and 34%.Source: Table 12 · p. 26, §F.1 · p. 26

F.2 ROUTING BETWEEN MODELS

  • Under model alternation, ReFold resolves 37 tasks at both loads against base’s 33 and 27, while cutting tokens by 59% and 27% and duration by 26% and 33%.At w=32, 13 alternating and 21 27B-only base tasks time out, against none and 2 for ReFold.Source: Table 13 · p. 26, §F.2 · p. 26

F.3 RETRIEVAL MEMORY

  • With retrieval memory at a 16k budget, adding ReFold raises resolved tasks from 8 to 17 and reduces budget exhaustions from 39 to 23.ReFold alone resolves 21 tasks and exhausts the budget on 14; the passage attributes its advantage over the combined setup to retrieved chunks refilling the freed budget.Source: Table 14 · p. 27, §F.3 · p. 27
Loading 2610.07863v1…