PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Coding Agents Sustain Continual Evolution without Forgetting?

Shangziqi Zhao, Yitong Zhang, Jia Li

arXiv:2611.03219v1cs.SEcs.AI

TL;DR

Prior harness self-evolution work has largely focused on individual task types, leaving open whether coding-agent harnesses can evolve across tasks while retaining earlier capabilities. The paper evaluates two methods across three software-engineering datasets and six evolution directions, finding substantial cross-task forgetting and identifying harness-level conflicts that contribute to it.

  • Problem

    Prior harness self-evolution studies primarily examine single task types, leaving open whether harnesses can evolve across multiple tasks without losing earlier capabilities.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates AHE and Meta-Harness across three software-engineering datasets and six cross-task evolution directions, then analyzes agent trajectories and harness modifications.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Both methods often improve individual tasks but substantially degrade performance when evolution spans different tasks, with verification, workflow, and budget-strategy conflicts identified as contributors.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Continual harness self-evolution remains challenging across diverse software-engineering tasks, and the identified conflicts motivate attention to conflict-aware evolution methods.

    Source: §1 · p. 2, §6 · p. 9

    The paper frames preservation of performance on previously encountered datasets as an ideal objective for continual evolution.

    Source: §2 · p. 3

Abstract

from arXiv · show

Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters. We find that existing approaches largely study agent evolution on a single task. In practice, coding agents may need to handle multiple task types over time. Whether an agent harness can continuously evolve across multiple tasks without suffering from catastrophic forgetting remains an important open question. In this paper, we investigate this open question through a comprehensive empirical study of continual harness self-evolution. We organize three software engineering task datasets into six different evolution directions and apply two representative harness evolution methods to perform continual evolution along these directions. Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets. For example, with DeepSeek-V4-Flash-0731, Meta-Harness improves its performance on SWE-Bench to 94.0% after evolution. However, after further evolution on FeatureBench, its performance on SWE-Bench drops to 32.0% by the fifth round, corresponding to a 66% relative decrease. Finally, based on our experimental findings, we identify several insights, providing directions for the continual self-evolution of agent harnesses. Code is available at https://github.com/THU-Agent/continual-harness-evolution.

1 Introduction

Harness self-evolution can improve coding agents without changing foundation-model parameters, but prior work has mainly studied single-task evolution. This study examines continual evolution across task types, finding substantial forgetting and identifying harness-level conflicts as contributors.

1 Introduction

  • A coding agent combines a foundation model with a harness that coordinates model, tool, and environment interactions, and harness choice can change performance substantially.With the foundation model fixed, different harnesses score 82.0% to 96.0% on SWE-Bench.Source: §1 · p. 1
  • Existing harness self-evolution studies primarily target one task type, although coding agents may need to handle varied tasks over time.The paper names this successive evolution across task types continual harness self-evolution.Source: §1 · pp. 1–2
  • The study evaluates two representative evolution methods in a continual setting and asks whether they retain earlier capabilities while improving on new tasks.The methods are AHE and Meta-Harness, evaluated on SWE-Bench, SWT-Bench, and FeatureBench.Source: §1 · p. 2
  • Continual evolution across different task types can substantially degrade previously acquired performance; Meta-Harness improves SWT-Bench to 100.0%, but after FeatureBench evolution SWT-Bench falls to 62.0%.The study evaluates AHE and Meta-Harness across three software-engineering datasets and six cross-task evolution directions.Source: §1 · p. 2
  • Manual analysis of more than 520 trajectories identifies verification, workflow, and budget-strategy conflicts as contributors to catastrophic forgetting.Ablations indicate that the conflict types have substantially different effects on performance.Source: §1 · p. 2

2 Problem Definition

Continual harness self-evolution updates a harness over an ordered sequence of task datasets using trajectories from each dataset. The central requirement is adapting to new tasks while limiting degradation on earlier ones, which the paper calls catastrophic forgetting when degradation is substantial.

2 Problem Definition

  • Continual harness self-evolution repeatedly updates a harness using trajectories from an ordered sequence of datasets, each representing a software-engineering task type.The update after dataset D_i uses trajectories collected from tasks in D_i.Source: §2 · p. 3
  • The central challenge is adapting to a new task without substantially degrading performance on previously encountered tasks.The paper relates this challenge to the stability–plasticity dilemma in continual learning.Source: §2 · p. 3
  • The paper defines catastrophic forgetting as substantial source-task performance degradation after further harness evolution on a target task.Forgetting is quantified using absolute and relative performance drops on the previous dataset.Source: §2 · p. 3
  • The ideal objective combines improvement on each new dataset with retention of performance on every previously encountered dataset.Eval denotes performance with a given harness and fixed foundation model; the two conditions capture adaptation and retention.Source: §2 · p. 3

3 Experimental Setup

The experiments study continual evolution among code repair, test generation, and feature implementation tasks using a fixed agent framework and foundation model. They compare two evolution methods under controlled implementation and execution settings.

3 Experimental Setup

  • The experimental setup evaluates continual evolution across code repair, test-case generation, and code-generation tasks.The section introduces datasets, the agent framework, evolution methods, and implementation details.Source: §3 · p. 3, §3.1 · p. 3

3.1 Datasets

  • The three datasets cover issue repair, test generation for real-world issues, and feature implementation with both new-feature and regression tests.Self-evolution uses 50 SWE-Bench Verified instances, 50 SWT-Bench instances, and 30 FeatureBench instances because of resource constraints.Source: §3.1 · pp. 3–4

3.2 Initial Agent Harness

  • Mini-SWE-Agent is the seed harness, with evolution restricted to system prompts, execution logic, and tools.The framework's interface specifications and execution environment are held outside the evolver's modification scope.Source: §3.2 · p. 4

3.3 Evaluated evolution algorithms

  • AHE analyzes successful and failed trajectories before modifying the harness, whereas Meta-Harness maintains a Pareto frontier and evaluates proposed modifications on the current dataset.The study implements both methods for Mini-SWE-Agent based on their original codebases and aligns settings with the original papers as closely as possible.Source: §3.3 · p. 4

3.4 Implementation Details

  • Experiments use DeepSeek-V4-Flash-0731, E2B, a one-hour task limit, and 9 AHE or 5 Meta-Harness evolution rounds.The rollout temperature is 0; each round executes the same task batch twice for trajectory analysis.Source: §3.4 · p. 4

4 Experimental Results

Single-dataset evolution improved benchmark scores, but continual evolution usually degraded prior-task performance. The analysis identifies three conflict types with unequal effects on that degradation.

4.1 RQ1: Catastrophic Forgetting During Continual Harness Self-Evolution

  • On individual datasets, AHE improved SWE-Bench from 82.0% to 96.0%, while both methods improved SWT-Bench from 82.0% to 100.0%.Both methods also improved FeatureBench by more than 10 percentage points.Source: Figure 2 · p. 5, §4.1 · pp. 5–6
  • Across six directions, almost every continual evolution path substantially degraded earlier-task performance, with forgetting varying by dataset and evolution direction.Degradation was smaller between SWE-Bench and FeatureBench than between SWT-Bench and either of the other datasets.Source: Figure 3 · p. 5, §4.1 · p. 6

4.2.1 Categorization of Harness Conflicts

  • Among pass-to-fail cases, verification conflicts accounted for 87.1%, workflow conflicts 8.2%, and budget conflicts 0.6%.The analysis examined harness code differences and 520 execution trajectories.Source: §4.2.1 · p. 7
  • Verification, workflow, and budget conflicts respectively alter task checks or artifacts, solution procedures, and time or computational-resource management.Artifact-specific constraints can impose the wrong submission type or gate for a previously encountered task.Source: §4.2.1 · p. 7

4.2.2 Impact and Case Study of Different Harness Conflict Types

  • 0.0 performance on SWE-Bench followed introduction of a mandatory artifact constraint suited to SWT-Bench but incompatible with source-code fixes.The constraint rejected solutions modifying files beyond the test files.Source: §4.2.1 · p. 7, Table 1 · p. 8, §4.2.2 · p. 8
  • Workflow modifications reduced SWE-Bench performance from 96.0 to approximately 66.0 and increased average task execution time by 235%.The FeatureBench-specific workflow blocked access to external resources and prompted unnecessary exploratory operations.Source: §4.2.2 · p. 8
  • Budget-strategy modifications had little impact on task performance but increased execution time by 79%.The study describes their main effects as changes in execution efficiency.Source: §4.2.1 · p. 7, §4.2.2 · p. 8

5 Related Work

Prior work evolves harness components or whole harnesses, but chiefly evaluates single task types. Continual learning research has focused more on memory-augmented agents than on joint evolution of the harness.

5.1 Harness Self-Evolution

  • Harness self-evolution research modifies components such as prompts, memory, and skills, or optimizes the harness as a whole.These studies primarily evaluate improvement on a single type of task, not continuous evolution across task types.Source: §5.1 · p. 9

5.2 Continual Learning for Agents

  • Continual-learning benchmarks and methods for agents have primarily considered skill accumulation, memory, task ordering, or experience accumulation.Relatively little attention has been paid to continual evolution of the harness as a whole, with multiple components jointly modified from execution experience.Source: §5.2 · p. 9

6 Conclusion

The study finds that continual harness evolution can cause catastrophic forgetting, with verification, workflow, and budget-strategy conflicts contributing to it.

6 Conclusion

  • Across three software engineering benchmarks and six cross-task evolution directions, performance often degraded substantially during continual evolution, with verification, workflow, and budget-strategy conflicts contributing to forgetting.The findings identify reliable evolution across diverse software engineering tasks as a challenge.Source: §6 · p. 9

A Implementation Details

Table 2 is identified as the configuration reference for reproducibility.

A Implementation Details

  • Table 2 presents configuration information for reproducibility.Source: Table 2 · p. 13

B Discussion and Case Study for Dataset Settings

The authors examine whether paraphrasing SWT-Bench prompts helps harness evolution rely on task characteristics rather than fixed wording. With original prompts, Meta-Harness adds a submission-artifact branch, but its condition is tied to the template.

B Discussion and Case Study for Dataset Settings

  • The experimental setting paraphrases SWT-Bench’s shared fixed linguistic pattern while preserving the prompts’ original semantics.The section discusses and validates the rationale for this setting.Source: B Discussion and Case Study for… · p. 13
  • With original SWT-Bench prompts, Meta-Harness adds a branch that requests source code for code-modification tasks and a test file otherwise.The authors describe this distinction as a positive signal for preserving submission requirements across task types.Source: B Discussion and Case Study for… · p. 13
  • The branch condition is built entirely from SWT-Bench’s fixed linguistic template, tying it to prompt surface form rather than task semantics.The authors characterize this evolution pattern as unfavorable for generalization.Source: B Discussion and Case Study for… · p. 13
  • After SWT-Bench prompt paraphrasing, self-evolution no longer produced similar template-dependent branches, suggesting paraphrasing can mitigate exploitation of dataset-specific wording.The authors say this encourages the evolved harness to rely more on task-level characteristics.Source: B Discussion and Case Study for… · p. 13
Loading 2611.03219v1…