PaperlayerView exact revision on arXiv

Source-linked AI summary

DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents

Tianle Chen, Pushkar Singh, Ketan Todi, Biswajit Paria, Qijia He, Wenqi Wang, Yuwen Tan, Jike Zhong, Sugato Basu, Deepti Ghadiyaram, Arjun Akula

arXiv:2610.19376v1cs.AI

TL;DR

Long-horizon agents can repeat tool calls and receive observations they already have, and history compression makes loops more common. DeLoop mines an agent’s own looping action and later recovery as a preference pair. On CodeGym, it reduces Qwen3-8B loops from 57.3% to 6.1% while using about one third as many steps per task, and raises Qwen3-32B task success from 37.7% to 44.8%.

  • Problem

    Long-horizon agents can repeat a tool call with the same arguments and get an observation they already have, adding no new information; compression makes loops more common.

    Source: §1 · p. 1

  • Method

    DeLoop uses a looping action as a negative and the first later different, non-looping action in the same rollout as a positive, training on the pair with DPO.

    Source: §1 · pp. 1–2

  • Results

    DeLoop cuts Qwen3-8B loop rate 9.4×, from 57.3% to 6.1%, at about one third as many steps per task while maintaining task success, and raises Qwen3-32B success from 37.7% to 44.8%.

    Source: §1 · p. 2, §5.2 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    An agent’s own loops can supply preference supervision without task rewards, gold answers, or LLM judges.

    Source: §1 · p. 2

Abstract

from arXiv · show

Language-model agents that run long tasks often redo work they have already done: they re-issue a tool call they made earlier and get back exactly the same observation, a step that adds no new information. We identify this failure as a loop. Loops appear natively in Qwen, Llama, and Gemini agents and become far more common once the agent's history is compressed to fit its context window. To reduce these loops, we propose DeLoop, a self-supervised training method. DeLoop builds on a simple observation: the agent's own rollout can provide the supervision needed to learn from its loops. We use a looping action as a negative example and the first later non-looping action in the same rollout as a positive example. DeLoop learns from these pairs through direct preference optimization and does not require task rewards, gold answers, or LLM judges. On CodeGym under compression, DeLoop reduces the share of Qwen3-8B tasks containing loops from 57.3% to 6.1%. It also uses about one third as many steps per task, with similar task success. On Qwen3-32B, DeLoop reduces loops and steps while raising task success from 37.7% to 44.8%.

1. Introduction

Long-horizon agents can repeat actions and receive no new information, a behavior made more common by compressed histories. DeLoop uses those loops as preference supervision and reduces looping while lowering interaction cost.

1. Introduction

  • On CodeGym, Qwen3-8B loop rate rises from 29.0% with full history to 57.3% under summarization; 59.5% of looping runs fail versus 33.4% of loop-free runs.The failure association was observed across six Qwen3 and Gemini models, and looping actions per task increase more than 3.5× under summarization.Source: §1 · p. 1
  • Prior context-management work asks what history to retain, whereas this paper studies whether the policy uses retained history to avoid repeating actions.The paper frames the complementary question of whether a recorded action helps the agent avoid repeating it.Source: §1 · p. 1
  • Within one rollout, DeLoop pairs a looping action as negative with the first later different, non-looping action as positive, then trains with DPO without external rewards, gold answers, or judges.The loop is identifiable from the action-observation trace without post-hoc failure attribution.Source: §1 · pp. 1–2
  • DeLoop cuts Qwen3-8B loop rate 9.4×, from 57.3% to 6.1%, with lower interaction cost and similar success; Qwen3-32B success rises from 37.7% to 44.8%.On Qwen3-14B, task success rises from 45.1% to 52.3%.Source: §1 · p. 2
  • The paper identifies loops even with full history, finds them more common under compression, and reports an association with task failure.A loop repeats an earlier action and receives the same observation.Source: §1 · pp. 1–2

2. Related work

Prior work addresses repetition through inference-time interventions, context management, and externally supervised training. DeLoop instead trains the policy from its own later recovery in the same rollout.

Repeated actions in agents

  • Reflection, early stopping, and blockers address repetition without changing policy; on CodeGym, a blocker leaves repeated-call proposals at 71.3% versus 69.8% without it, while DeLoop lowers them to 13.6%.The cited blocker comparison is for Qwen3-8B.Source: §2 · pp. 2–3

Context management

  • Context-management methods store, summarize, or learn what history to retain, while this work asks whether agents use retained history to avoid repeating actions.The paper also notes that removing plans from context sharply hurts performance.Source: §2 · p. 3

Learning from agent trajectories

  • Many trajectory-training methods use expert actions, outcome feedback, step labels, rewards, or critiques; DeLoop instead uses the agent’s later recovery and needs none of those labels.Its label is the exact-repeat rule, and it also learns from failed rollouts.Source: §2 · p. 3

Penalizing repetition

  • Direct repetition penalties mark actions to avoid, whereas DeLoop also supplies an alternative: loop-flag penalties yield a 15.0% loop rate versus 6.1% for DeLoop on Qwen3-8B.The comparison is on CodeGym.Source: §2 · p. 3

3. Loops in Long-Horizon Agents

A loop is a repeated tool call that returns the same observation, adding no new information. Loops cost extra actions and tokens, are associated with higher failure rates, and become more common under compression.

3.1. Defining a Loop

  • A trajectory loops if it contains at least one repeated step, where a looping action repeats a prior call and receives the same observation.A repeated call that returns a new observation is not a looping action.Source: §3.1 · p. 3

3.2. Loops Are Costly

  • Looping Qwen3-8B runs average 6.5 looping actions and 10.5 more actions per task than loop-free runs, with 11k more transcript tokens.The comparison is for CodeGym under summarization at 4,096 tokens; the same cost pattern holds across all six models.Source: §3.2 · p. 4
  • 85% of looping Qwen3-8B runs fail versus 68% of loop-free runs; Gemini 3.1 Pro shows 48% versus 12%.Looping runs are more likely to fail than loop-free runs across all six evaluated Qwen3 and Gemini models.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Compression Makes Loops More Common

  • Qwen3-8B’s loop rate rises from 29.0% with full history to 57.3% under summarization at 4,096 tokens and 64.7% at 2,048 tokens.All three Gemini models also show higher loop rates with compression; Gemini 3.1 Pro has the largest increase, at 2.9×.Source: §3.3 · p. 4
  • Qwen3-8B repeats a recent call in 3.3% of samples when its record remains in context, versus 48.8% when that record is removed.Keeping the record does not prevent episode-level loops: the full-history loop rate is 29.0%, and under compression 25.2% of runs repeat a call still present word for word.Source: §3.3 · p. 4
  • Action-record summaries still loop in 52.0% to 62.7% of runs, while ACON and LLMLingua-2 reach 46.3% and 46.7%, above the 29.0% full-history rate.The agent can fail to use a record it has, motivating training the policy itself.Source: §3.3 · p. 4

4. Repeated Actions as Self-Supervision

DeLoop uses an agent’s own loop and subsequent recovery as preference supervision. It mines pairs from rollouts and trains the policy with direct preference optimization.

4. Repeated Actions as Self-Supervision

  • DeLoop treats a looping action as rejected and the first later different, non-looping action in the same rollout as chosen.Both actions are paired with the history preceding the loop; a rollout that never recovers contributes no pair, while one rollout can contribute multiple pairs.Source: §4 · p. 4, §4.1 · p. 5, Figure 4 · p. 5

4.1. Mining Preference Pairs from Loops

  • Pair construction uses actions and observations alone, so recovery supplies a training pair even when the rollout ultimately fails, without task rewards, gold labels, or an LLM judge.The chosen action is the agent’s first later loop-breaking action in that rollout.Source: §4.1 · p. 5

4.2. Preference Training

  • DeLoop trains action preferences with DPO; Qwen3-8B uses one epoch of LoRA DPO (β = 0.05) on 8,000 self-mined CodeGym pairs.At inference, the trained policy generates each next action; the same updated model summarizes older interactions under a separate prompt without a separate summary-generation objective.Source: §4.2 · p. 5

5. Experiments

The experiments evaluate DeLoop on CodeGym and test its performance across model sizes, training objectives, and cross-domain data. The results show fewer loops and lower interaction costs, with task success maintained or improved in several settings.

5.1. Experimental Setup

  • The evaluation uses 300 held-out CodeGym tasks under 4,096-token summarization by default, alongside full-history reference runs and baselines including context methods, inference controls, and trained policies.The experiments also evaluate Qwen3-14B and Qwen3-32B on CodeGym and use cross-domain pairs for DeLoop-Mix.Source: §5.1 · p. 6

5.2. Main Results on CodeGym

  • 57.3% to 6.1%: DeLoop cuts Qwen3-8B’s CodeGym loop rate, while steps per task fall from 15.00 to 4.67 and tokens per task fall by 47%, with task success holding.The comparison uses summarization at 4,096 tokens.Source: §5.2 · pp. 6–7, Table 1 · p. 6
  • At larger scales, loops fall 3.5× on Qwen3-14B and 3.0× on Qwen3-32B, tokens per task fall by 35% and 29%, and task success rises by 7.2 and 7.1 points, respectively.Figure 5 compares task success with loop rate, steps, and estimated tokens per task across model sizes.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3. Ablation Studies

  • Imitating only the loop-breaking action leaves a 37.7% loop rate, versus 6.1% when paired against the looping action; DPO, KTO, and IPO all reduce loops to 15.0% or below.DPO achieves the lowest loop rate and highest task success among the tested objectives; repeated actions returning new information rise from 29.9% to 69.8%.Source: §5.3 · p. 7, Table 2 · p. 7

5.4. Effect of Cross-Domain Training Data

  • With 4,635 additional cross-domain pairs, DeLoop-Mix raises compressed Qwen3-8B CodeGym success from 25.1% to 30.3% and full-history success from 31.7% to 36.2%.Under compression, the loop rate falls from 57.3% to 10.1%; Figure 6a compares success with and without compression.Source: §5.4 · p. 8, Figure 6 · p. 8

6. Robustness and Generalization

The paper tests whether DeLoop’s loop reduction persists across context settings, memory policies, and agent environments. It reports lower loop rates across these settings, with some evaluations also showing lower interaction costs or higher task success.

6.1. Generalization Across Compression Settings and Memory Policies

  • Across five CodeGym context settings, DeLoop’s loop rate stays between 4.7% and 12.8%, while the untrained baseline reaches 65.3%.The settings range from full history to summarization at 2,048 tokens and truncation at 4,096 tokens.Source: §6.1 · p. 8
  • Under all seven alternative memory policies, DeLoop-Mix lowers loop rate, steps, and tokens; with an action ledger, loop rate falls from 62.7% to 11.3% and task success rises by 5.7 points.The tested policies range from retrieval memory and Claude-style compaction to summaries augmented with an action ledger.Source: §6.1 · p. 8

6.2. The Improvement Extends to Other Agent Environments

  • Across ALFWorld, Open-AgentRL, AppWorld, and ScienceWorld, loops fall in all four environments, by up to 32.8 points on AppWorld; Open-AgentRL success rises from 43.9% to 45.5%.ALFWorld and Open-AgentRL use unseen games disjoint from the training games behind their pairs.Source: §6.2 · p. 8

6.3. The Repair Transfers to New Benchmarks

  • In native harnesses, DeLoop reduces Terminal-Bench duplicate calls from 16.6 to 6.9 per task, while DeLoop-Mix lowers BrowseComp-Plus loop rate from 22.4% to 1.2% with accuracy matching the untrained model.Zero-shot CodeGym-trained Qwen3-32B also lowers ALFWorld loop rate from 64.9% to 54.5% and raises task success from 33.6% to 42.5%.Source: §6.3 · p. 9

7. Conclusion

The conclusion presents DeLoop as training from an agent’s own loops and later loop-breaking actions. On CodeGym, it reduces loops and steps while maintaining task success at 8B, and raises task success at 14B and 32B.

7. Conclusion

  • DeLoop uses an agent’s own loop as a rejected action and its later loop-breaking action as the chosen one, reducing dependence on external supervision.The conclusion reports a 9.4× loop-rate reduction on Qwen3-8B, from 57.3% to 6.1%, with about one third as many steps per task and maintained task success.Source: §7 · p. 9

A. Evaluation Details

The appendix describes evaluation-run averaging and the measures used for interaction cost and loops. A rollout example shows how a repeated search can be paired with a later action that advances the task.

A. Evaluation Details

  • Most CodeGym results use 300 held-out tasks under 4,096-token summarization, but the number of runs varies by method and comparison.For example, Table 1 averages three runs for the full-history agent, compressed baseline, DeLoop, and DeLoop-Mix; many other rows use one run.Source: A. Evaluation Details · p. 14
  • Steps count executed actions, tokens are estimated from logged trajectories, and loop measurement uses CodeGym’s structural detector or each environment’s native repeat detector.Open-AgentRL does not record tokens, so its Table 3 token cells are empty.Source: A. Evaluation Details · pp. 14–15

B. A Real Loop from CodeGym

  • In the rollout example, three identical searches return the same cell before a later MarkVisited action; DeLoop treats the repeated search as looping and MarkVisited as loop-breaking.Loop-breaking actions can also be final answers, as in a later Done(answer=1) action paired against a repeated FindUnvisitedLand() call.Source: B. A Real Loop from CodeGym · p. 15

C. More Compression Methods and Model Sizes

Compression worsens looping across the tested Gemini compression methods, while DeLoop reduces looping across Qwen model sizes. The action-record probes find that preserving the record matters more than its wording, and training reduces re-issuing even when the record is deleted.

C. More Compression Methods and Model Sizes

D. Setup Details

  • A looping action is a canonicalized tool call seen earlier in the episode that returns a byte-identical observation.The standard compression threshold is 4,096 tokens; when exceeded, the harness retains the system prompt, task, and six most recent messages and summarizes the intervening history.Source: D. Setup Details · p. 15

F. How the Policy Uses Its Action Record

  • Deleting the action record raises re-issue from 15.3% to 43.3%, while changing its wording keeps re-issue near the verbatim rate of 15.3%.The tested variants were the original action–observation block, a one-line assertion, no record, the assertion in the summary, and the returned value without stating that the action was run.Source: F. How the Policy Uses Its Action Record · p. 17
  • On the same points, training lowers re-issue in every condition: DeLoop's rates are 3.4% with the verbatim record, 4.5% with the assertion, and 15.0% with record deleted, versus 15.3%, 17.6%, and 43.3% for the compressed baseline.Source: F. How the Policy Uses Its Action Record · p. 17
  • In an unselected sample of 300 post-compression decision points, the untrained policy re-issues the last action 3.3% of the time with its record present and 48.8% with it deleted.Weighting the loop-stratified groups by their population shares gives a 44.3-point difference.Source: F. How the Policy Uses Its Action Record · p. 17

F.1. Training Suppresses the Looping Action

DeLoop trains on within-rollout pairs contrasting a looping action with a later loop-breaking action, mined from actions and observations without task labels. Across training budgets, it reduces loop rates more than matched baselines, and more self-mined pairs coincide with fewer loops and steps.

F.1. Training Suppresses the Looping Action

  • At held-out loop points, DeLoop suppresses the looping action rather than teaching one memorized recovery: it re-issues less often, while copying the recorded loop-breaking action stays between 18.8% and 20.8%.Calls to Done() rise from 3.6% to between 12.4% and 26.0%.Source: §F.1 · pp. 17–18

G.1. Training Details

  • The training pair uses the looping action as rejected and the first later action that is neither looping nor the same call as chosen.Pairs are mined by comparing canonicalized calls and exact observations; the additional condition avoids choosing a later repeat that returns a new observation but has the same action text.Source: §G.1 · pp. 18–19
  • The miner reads only actions and observations, not rewards, task status, or gold answers, and uses no LLM judge.The 8,000 CodeGym pairs include 82.6% from runs that fail.Source: §G.1 · pp. 18–19

G.2. Matched Baselines

  • At 3,910 and 8,000 examples, DeLoop has the lowest loop rate (18.6% and 6.1%) and fewest steps per task; gold-label DPO reaches 37.5% and 22.3%.The 8,000-example rows are also reported in Table 1.Source: §G.2 · p. 19, Table 10 · p. 19

G.3. More Pairs, Fewer Loops

  • The loop rate falls from 45.1% at 1,000 self-mined pairs to 6.1% at 8,000, while steps per task fall from 11.88 to 4.67.The loop-rate reduction levels off after 8,000 pairs.Source: §G.3 · p. 19

G.4. How Much Cross-Domain Data to Add

Across additional settings, DeLoop reduces loops while retaining useful activity, and mixed pairs and public trajectories extend the gains beyond CodeGym. The analyses attribute the training effect to contrasting both actions in a pair, while identifying context ambiguity as a constraint on activity.

G.4. How Much Cross-Domain Data to Add

  • At a 37% cross-domain share, three data sources lower the loop rate from 19.0% to 10.1%, while task success is 30.3% versus 30.9% for the two-source mixture.Every tested mixture raises success above the compressed baseline’s 25.1% and DeLoop’s 25.9%; loop rate remains at or below 24.3% through a 50% cross-domain share.Source: §G.4 · pp. 19–20, Table 12 · p. 20

G.5. Comparison with Agent-R-Style Splicing

  • With 1,000 pairs, self-mined training yields a 45.1% loop rate, 11.88 steps per task, and 26.4% task success, versus 52.3%, 13.62, and 24.0% for Agent-R-style splicing.The self-mined pairs do not use outcome labels.Source: §G.5 · p. 20, Table 13 · p. 21

G.6. Hyperparameters, Mining Rules, and Model Size

  • DeLoop lowers loops across model sizes and sources: Qwen3-14B trained on Qwen3-8B pairs falls from 42.9% to 9.3%, while Llama-3.1-8B falls from 43.7% to 2.0% on its own pairs.For Qwen3-32B, training on its own pairs wins back 7.1 of the 9.0 task-success points lost to compression; its full-history evaluation also improves loop rate from 25.0% to 11.7% and success from 46.3% to 51.3%.Source: §G.6 · p. 20, §G.17 · pp. 24–25

G.7. Per-Step Selectivity

  • DeLoop reduces looping actions from 24.9% to 2.5% of steps and informative repeats from 10.6% to 5.8%, raising selectivity from 29.9% to 69.8%.DeLoop-Mix reaches 74.2% selectivity; informative repeats return new observations, unlike looping actions.Source: §G.7 · p. 20, Table 17 · p. 22

G.8. Training Objectives on the Same Loop Signal

  • Across objectives trained on the same pairs, DPO reaches 4.67 steps per task and IPO 6.72; KTO reduces the loop rate to 12.6%.All four objectives shorten episodes and reduce tokens per task from 23.9k to between 12.4k and 13.6k.Source: §G.8 · pp. 20–21

G.9. Inference-Time Controls

  • Every inference-time control leaves the loop rate at 46.7% or above, versus 6.1% for DeLoop; the duplicate blocker barely changes steps per task, from 15.00 to 14.71.The blocker’s low step reduction occurs because the agent keeps proposing the blocked call.Source: §G.9 · p. 22

G.10. GRPO Variants

  • The per-step loop penalty cuts loop rate to 15.0%, while a sparse episode-level loop reward leaves 47.3% looping runs and loop-weighted SFT leaves 42.3%.Gold-action GRPO has only 2.7% informative groups with exact-match rewards and a 52.7% loop rate; partial credit and diverse sampling raise informative groups and lower the loop rate.Source: §G.10 · p. 22

H.3. Why Both Sides of a Pair Matter

  • The DPO pair raises the loop-breaking action and lowers the looping action under the same context; with an α = 1 likelihood anchor, the loop rate rises from 6.1% to 27.0%, and chosen-only SFT reaches 37.7%.The analysis says the anchor reduces the contrast between preferred and rejected actions; a stricter chosen-action rule also performs worse than the standard rule at the same pair count (41.7% vs. 38.7% loop rate).Source: §G.11 · p. 22, §H.3 · p. 26

I.1. The Policy Reacts to the Record of Its Own Executed Action

These analyses examine how action records affect re-issuing, which repetition patterns compression amplifies, and whether loop-specific preference pairs matter. They also test DeLoop against repeats of summarized calls and a duplicate blocker, using analyses that require no new rollouts.

I.1. The Policy Reacts to the Record of Its Own Executed Action

  • Execution records reduce re-issuing and help distinguish loops from retries; trained DeLoop policies still avoid most loops when the earlier record is deleted.For untrained Qwen3-8B, re-issue falls to 16.5% with the record, but rises to 53.3% when it is deleted; at retry points, the record lowers re-issue from 68.4% to 51.7%, while a stated observation lowers it to 20.6%.Source: §I.1 · pp. 26–27

I.2. Compression Amplifies Exact Repetition Most

  • Under summarization, exact-loop actions rise 3.77×, from 6.6% to 24.8%, while broader repetition signatures change by only 0.73× to 1.21×; training also lowers near-duplicates.Near-duplicate calls fall from 34.3% of actions to 19.0% under DeLoop and 23.7% under DeLoop-Mix.Source: §I.2 · p. 27, Table 26 · p. 28

I.3. Random Pairs Do Not Break the Loop

  • At about 500 updates, 1,003 self-mined pairs reduce loop rate to 9.4%, while random pairs raise it to 69.5% versus the compressed baseline’s 66.1%.With 4,012 self-mined pairs, loop rate reaches 5.5%, matching DeLoop trained on all 8,000 pairs.Source: §I.3 · p. 27

I.4. The Loop-Breaking Action Matters

  • With the loop-breaking action, loop rate is 5.7%, versus 41.9% with a random history action and 36.1% with another pair’s chosen action.The controls retain the same prompts, rejected actions, and training recipe, varying only the chosen action.Source: §I.4 · p. 27, Table 28 · p. 28

J.1. Loops on Calls Still in Context

  • DeLoop lowers loop rates for calls still in context from 25.2% to 3.2% and for calls already summarized from 32.1% to 2.9%.The compressed agent’s added loops come from calls the harness summarized, despite instructions to preserve executed actions and results.Source: §J.1 · pp. 28–29, Table 29 · p. 29

J.2. The Duplicate Blocker

  • A duplicate blocker leaves repeat proposals nearly unchanged (71.3% versus 69.8%), while DeLoop lowers repeat-proposal tasks to 13.6% and steps per task from 15.00 to 4.67.The methods are complementary: with the blocker on, DeLoop runs a repeated call in 1.3% of tasks and uses 12.8k tokens per task.Source: §J.2 · p. 29, Table 30 · p. 29

J.3. Gemini Models on the Same Tasks

On matched CodeGym tasks where compression fired, all three Gemini models loop more often under compression. DeLoop also lowers looping among episodes of similar length and on a step-index measure, so the reduction is not just an effect of shorter episodes.

J.3. Gemini Models on the Same Tasks

  • On matched tasks where compression fired, Gemini 3.1 Pro loops in 61.9% of tasks with compression versus 33.1% without; the compression ratios are 1.4× for Gemini 3.1 Flash and 1.2× for Gemini 3.5 Flash-Lite.The corresponding ratios are 1.8×, 1.4×, and 1.1× when comparing full-history episodes long enough for compression to fire.Source: §J.3 · p. 29

J.4. Fewer Loops Are Not Just Shorter Episodes

  • Among episodes of similar length, loop rate falls from 33.2% to 13.1% for 6–10 steps and from 69.1% to 29.2% for 11–20 steps.Loop hazard also falls at every step index, including from 22.3% to 4.1% at steps 5–9 and from 32.8% to 7.0% at steps 10–14. On 164 tasks answered within four steps on average, DeLoop and the compressed baseline both solve 30.1% of runs, in 2.7 versus 13.0 steps.Source: §J.4 · p. 30, Table 32 · p. 30

Related papers

Loading 2610.19376v1…