PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness Evolution as Learning: Approximation, Generalization, and Optimization Limits of Self-Improving Personal Agents

Zeyu Gan, Zixuan Gong, Yong Liu

arXiv:2609.36892v1cs.AIcs.CL

TL;DR

Personal agents must accommodate requirements that vary across users and persist across tasks, while diverse harness designs and evolution algorithms create practical tensions. The paper evaluates harness mechanisms, memory scale, and update procedures on AppWorld-P, then analyzes harness evolution as learning with approximation, generalization, and optimization errors. Experiments find that preference compliance depends on harness mechanisms, does not improve monotonically with memory size, and can remain below oracle performance despite repeated updates.

  • Problem

    Personal agents must accommodate requirements that vary across users and persist across tasks, while diverse harness designs and evolution algorithms create practical tensions.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates harness mechanisms, memory scale, and update procedures on AppWorld-P, then analyzes harness evolution as learning with approximation, generalization, and optimization errors.

    Source: §1 · p. 2, §3.3 · p. 4, §4 · p. 5

  • Results

    Experiments find that preference compliance depends on harness mechanisms, does not improve monotonically with memory size, and can remain below oracle performance despite repeated updates.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Reliable personalization calls for jointly considering what a harness can express, how its capacity matches available evidence, and how feedback guides updates.

    Source: §5 · p. 10

    The study uses simulated AppWorld-P environments with explicitly specified preference rules; practical deployment also requires attention to privacy, safety, and user control.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

As the capabilities of large language models (LLMs) continue to advance, increasing attention is turning to how to translate their abilities into useful behavior. Personal agents bring this question into everyday settings, where models are expected to serve individual users and continually adapt to their preferences. With the underlying model held fixed, such adaptation relies on harness engineering: designing and evolving the surrounding layer that manages context, memory, tools, and execution. Despite rapid progress, the factors governing effective harness evolution remain insufficiently understood. To narrow this gap, we investigate three central questions concerning harness architecture, harness scale, and self-evolution algorithms through complementary empirical and theoretical analyses. Empirically, we introduce a preference-oriented benchmark and systematically characterize the capabilities and limitations of personal agents associated with these three dimensions. Theoretically, we formulate harness evolution as a learning problem and explain these phenomena through approximation, generalization, and optimization errors. Analyses of reachable policies, capacity under finite interaction evidence, and biased update dynamics provide theoretical accounts of the observed phenomena. Together, these results offer a unified perspective on the limits of personalization through harness evolution and inform future harness design.

1 INTRODUCTION

Personal agents must adapt to persistent user preferences while their foundation models remain fixed, making harness design and evolution central to reliable personalization. This paper studies three related tensions empirically and theoretically, organizing them through approximation, generalization, and optimization errors.

1 INTRODUCTION

  • Harness effectiveness depends on the preference: explicit context can suffice for local conventions, while calendar constraints may require state checks and reasoning.The introduction also notes that larger memories can interfere with outputs and that misinterpreted feedback can produce inaccurate or unnecessary updates.Source: §1 · p. 2, Figure 1 · p. 2
  • Existing work documents individual failures, but a standardized quantitative study of all three tensions and a common theoretical account remain lacking.This gap makes it difficult to determine whether to change harness mechanisms, scale, or evolution algorithms.Source: §1 · p. 2
  • The paper introduces a preference-oriented benchmark and a learning-theoretic framework linking harness architecture, memory scale, and update dynamics to approximation, generalization, and optimization errors.Its analyses address reachable policies, memory capacity and estimation, and residual error under biased updates.Source: §1 · pp. 2–3
  • The paper reports that explicit context handles some local preferences, computational mechanisms improve more demanding ones, memory expansion can yield non-monotonic compliance, and self-evolving recipes retain oracle gaps.These empirical findings are paired with analyses of reachability limits, finite-evidence capacity and estimation, and biased updates.Source: §1 · p. 3

2 RELATED WORKS

Related work covers personal-agent memory and preference adaptation, harness mechanisms for interacting with environments, and methods that evolve harness components. The paper situates its fixed-model study among these efforts.

Personal Agents

  • Personal-agent research studies persistent workspaces, memory as a substrate for agent behavior, preference following, and online reinforcement learning from user and environment responses.This paper instead fixes the foundation model and studies how a local harness learns to express preferences through interaction and feedback.Source: §2 · p. 3

Harness Engineering

  • Harness-engineering work spans reasoning-action interfaces, executable-code actions, structured software repair, context curation, persistent artifacts, verification, and environment-based evaluation.Source: §2 · p. 3

Self-Evolving Harness

  • Self-evolving harness methods modify feedback memory, prompts, playbooks, persistent context, action mechanisms, or harness construction, while analytical treatments are also emerging.The cited approaches include online adaptation, memory validity tracking, and oracle-gap decompositions.Source: §2 · p. 3

3 PRELIMINARY & PROBLEM FORMULATION

The paper models personalization as learning a local harness around a frozen foundation model. Its risk decomposition separates limits from reachable policies, finite interaction evidence, and the evolution algorithm.

3.1 PERSONAL AGENTS AND LOCAL HARNESS

  • The task combines a natural-language instruction with application state and tools, and the frozen model induces interaction traces while adaptation occurs outside its parameters.Source: §3.1 · pp. 3–4
  • A harness mediates between model and environment by constructing inputs and maintaining state; context harnesses inject text, while control harnesses can intervene in actions.Source: §3.1 · p. 4

3.2 HARNESS EVOLUTION AS LEARNING

  • The objective is to minimize user-distribution risk over harnesses, but the ideal user-preference policy is not directly observed and interaction feedback may be weak.Source: §3.2 · p. 4
  • An evolution algorithm updates the harness from interaction data, while the deployed harness can differ from the empirical-risk minimizer.Source: §3.2 · p. 4

3.3 ERROR DECOMPOSITION: APPROXIMATION, GENERALIZATION AND OPTIMIZATION

  • The risk gap from the user-optimal policy is decomposed into approximation, generalization, and optimization errors by comparing the reachable-class optimum, empirical minimizer, and learned harness.Source: §3.3 · pp. 4–5
  • Approximation error concerns the reachable policy class, generalization error the finite data and effective capacity, and optimization error the evolution algorithm.Source: §3.3 · pp. 4–5
  • Q1 compares context and computational or action-level control while fixing the model, actuator, and task distribution.Source: §3.3 · p. 5
  • Q2 studies memory-length sweeps under finite interaction evidence, while Q3 compares update recipes against oracle context across held-out checkpoints.Controlled AppWorld-P experiments are followed by theoretical analyses under explicit modeling assumptions.Source: §3.3 · p. 5

4 MAIN RESULTS

The main results use AppWorld-P to examine harness capability, memory scale, and self-evolution, interpreting these dimensions through approximation, generalization, and optimization errors.

4 MAIN RESULTS

  • The evaluation uses AppWorld-P, a preference-oriented extension of AppWorld, and measures the fraction of applicable episode–rule pairs that violate a preference.Harness-evolution conditions are evaluated on held-out tasks after the harness is frozen at each checkpoint.Source: §4 · p. 5

4.1 APPROXIMATION: ON THE CAPABILITY BOUNDARIES OF CONTEXT HARNESS

  • Context handles some in-support preferences, while computation or action rewriting substantially improves compliance on out-of-support preferences.The theoretical analysis explains this boundary by placing context-reachable policies within the pretrained model’s supported policy class, while control can transform executed traces.Source: Figure 3 · p. 6, §4.1 · pp. 6–7
  • The approximation analysis says refining context alone cannot eliminate the gap when the user-optimal policy is reachable through control but not context.The result is conditional on the stated risk separation and reachability assumptions.Source: §4.1 · pp. 6–7

4.2 GENERALIZATION: ON THE IMPACT OF HARNESS SCALE

  • Violations fall from approximately 0.77 at L = 0 to 0.20 at L = 10, then rise to approximately 0.25–0.27 at L = 20–150.The same decline-and-rise pattern remains when the evaluation episodes are re-scored on six preferences.Source: §4.2 · p. 7, Figure 4 · p. 7
  • With a fixed feedback budget, expanding memory divides observations across more statements, increasing estimation error even as coverage grows.Once coverage saturates, further expansion no longer reduces approximation error, while estimation error can continue to increase.Source: §4.2 · pp. 7–8

4.3 OPTIMIZATION: ON THE LIMITS OF SELF-EVOLVING HARNESS

  • ACE and TEPA finish near 0.48, leaving an approximately 0.40 gap to the 0.071 oracle rate; TRACE ends near 0.57 and Reflexion stays near the 0.881 no-memory reference.All tested self-evolution procedures, including full-rewrite and corrective-feedback diagnostics, finish above the oracle within the evaluated interaction budget.Source: Figure 5 · p. 8, §4.3 · p. 9
  • Under the stated assumptions, persistent biased updates can leave a positive optimization floor even as risk-estimation error vanishes.The lower bound applies across update rules sharing the specified constants, although their trajectories and limiting risks may differ.Source: §4.3 · p. 9

5 CONCLUSION

The paper finds that harness mechanisms, memory scale, and repeated updates shape personalization around a frozen foundation model. Its learning-theoretic framework connects these limits to approximation, generalization, and optimization errors.

5 CONCLUSION

  • Experiments on AppWorld-P show that compliance depends on available harness mechanisms, memory expansion is not monotonically beneficial, and repeated updates can leave performance below oracle.These findings concern personalization around a frozen foundation model.Source: §5 · p. 9
  • The framework offers conditional explanations for when and why adaptation falls short through distinct approximation, generalization, and optimization limitations.It links the observed limits to what a harness can express, how capacity matches available evidence, and how feedback guides updates.Source: §5 · pp. 9–10
  • The unified account supports failure diagnosis and more principled design by identifying which error components harness-engineering practices seek to reduce.The paper concludes that reliable personalization requires jointly considering expressivity, evidence-matched capacity, and feedback-guided updates.Source: §5 · p. 10

AI USE STATEMENT

The authors report using Codex and Claude Code across implementation, data, evaluation, analysis, writing, and figure preparation, while retaining responsibility for the final work.

AI USE STATEMENT

  • Codex and Claude Code assisted with implementation, data construction and processing, evaluation, interpretation, translation, mathematical checking, literature work, drafting, editing, and figures.The authors reviewed the AI-assisted work.Source: AI USE STATEMENT · p. 10
  • The authors developed the initial research ideas, conceptual framework, hypotheses, proof strategies, and experimental design without AI assistance.Source: AI USE STATEMENT · p. 10
  • The authors state that they take full responsibility for the final content, including AI-assisted text, references, mathematical arguments, code, data, evaluation procedures, results, and figures.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The study evaluates personal-agent adaptation in simulated application environments using explicitly specified preference rules. It flags privacy, unintended-action, and oversight concerns for practical deployment.

ETHICS STATEMENT

  • The study is limited to simulated application environments and explicitly specified preference rules.Its findings therefore concern this evaluation setting.Source: ETHICS STATEMENT · p. 10
  • Persistent memories may contain sensitive information, and incorrectly inferred preferences may lead to unintended actions.Source: ETHICS STATEMENT · p. 10
  • Responsible deployment requires user consent, protection of preferences and interaction logs, and oversight of consequential actions.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper describes where its experimental protocols and theoretical support are documented, then specifies the benchmark layer, model, actuator, and evaluation conventions. AppWorld-P adds preference rules and feedback to unchanged AppWorld task environments.

REPRODUCIBILITY STATEMENT

  • Appendix A documents experimental protocols, while Appendices B.1–B.4 provide assumptions and proofs supporting the theoretical claims.The code is open-sourced through an anonymous GitHub repository.Source: REPRODUCIBILITY STATEMENT · p. 10

A DETAILED EXPERIMENTAL SETTINGS

  • AppWorld-P attaches programmatic preference rules and episode feedback to unchanged AppWorld tasks, without exposing rule semantics or checker state beyond each harness arm’s channel.Source: A DETAILED EXPERIMENTAL SETTINGS · p. 13
  • All experiments use claude-haiku-4-5-20251001 with one literal-argument API call per step, a default 50-step task budget, and a frozen evaluation set.The constrained actuator prevents outsourcing arithmetic or aggregation to an interpreter.Source: A DETAILED EXPERIMENTAL SETTINGS · p. 13
  • The primary metric is the micro-averaged preference violation rate, and reported confidence intervals use Wilson 95% intervals.Source: A DETAILED EXPERIMENTAL SETTINGS · p. 13
  • Each persona is a declarative bundle of rules specifying a constrained API action, a checked field or state variable, and a context-arm natural-language statement.The checker determines applicability from recorded API calls and evaluates the resulting note or message.Source: A DETAILED EXPERIMENTAL SETTINGS · p. 13

A.1 DETAILED EXPERIMENTAL SETTINGS FOR Q1

Q1 tests whether harness mechanisms change which preferences a fixed model can satisfy, using empirically calibrated in-support and out-of-support rules. Its conditions contrast contextual specifications with checking, retries, and external computation.

Objective and preference selection

  • Q1 calibrates in-support rules as near-perfect under an explicit description but not without memory, and out-of-support rules as still substantially violated despite that description.These labels are determined empirically rather than from rule source files or prior judgment.Source: §A.1 · p. 13
  • The five preferences include two in-support instruction-following rules and three out-of-support rules involving exact computation, history-based inference, or aggregation.Source: §A.1 · p. 13

Harness conditions

  • The five conditions range from no memory and an explicit stated rule to history-derived context, checker rejection with retries, and preference-specific computation.The conditions compare mechanisms, not equal inference budgets.Source: §A.1 · p. 14
  • Checker rejects provides bounded rejection feedback without returning the value that must be computed, while Harness computes supplies derived values or rewrites action arguments.Source: §A.1 · pp. 14–15

Personas, streams, and sample sizes

  • The main persona uses eight training tasks, seven frozen evaluation tasks, six seeds, and a 50-step attempt budget; the crossover persona uses six training and six evaluation tasks, three seeds, and 60 steps.The habitual-card experiment uses a synthetic binary-feedback stream followed by six fixed AppWorld evaluation tasks.Source: §A.1 · p. 15

Scoring and controls

  • Scoring uses the same evaluation tasks across arms, but applicable-episode counts may differ; habitual-card scoring uses the first card attempted.For running total, only successful payments contribute to the cumulative amount or make the rule applicable.Source: §A.1 · p. 15

A.2 DETAILED EXPERIMENTAL SETTINGS FOR Q2

Q2 varies the number of learner-generated memory lines for eight preferences across messaging, Venmo, and Spotify. Assertions are collected from task trajectories and feedback, then selected to increase rule coverage as memory grows.

Objective and scan set

  • The Q2 sweep covers eight preferences across SMS, Venmo, and Spotify, with memory statements learned from instance-level feedback rather than hand-written oracle statements.The retained rules have learner-generated statements that could affect their corresponding checker.Source: §A.2 · p. 15

Learner-generated assertion pool

  • Assertions are collected over 32 episodes with empty injected memory, deduplicated, grouped by generating rule, consensus-ranked, and selected round-robin across rule groups.Consensus ranking uses generated text rather than oracle rule statements.Source: §A.2 · p. 15
  • The collection prompt receives the task, write actions, and episode-specific complaint, but not the rule name or oracle statement.Source: §A.2 · p. 15

Primary length sweep

  • The primary length sweep uses assertions about the eight scored preferences, with a finite pool recycled when memory length exceeds the number of distinct assertions.Selected lines are deterministically shuffled for each length and seed.Source: §A.2 · p. 16
  • Each sweep cell evaluates 120 episodes across 12 frozen tasks and 10 fresh rollouts per task, over three seeds.The task pool is balanced so each of the eight rules has four triggering tasks per cell before behavioral inapplicability.Source: §A.2 · p. 16

A.3 DETAILED EXPERIMENTAL SETTINGS FOR Q3

Q3 fixes the model, actuator, task distribution, and preference set to test how self-evolving harnesses approach an oracle context. Its shared feedback protocol compares an ACE-style updater with three alternative memory and enforcement mechanisms.

A.3 DETAILED EXPERIMENTAL SETTINGS FOR Q3

  • The five jointly satisfiable phone and Venmo preferences distinguish targets recoverable from feedback from literals that feedback withholds.The distinction is checked by comparing each oracle statement’s quoted literals with checker details available to the updater.Source: §A.3 · pp. 16–17, Table 3 · p. 16
  • Training uses 24 on-policy episodes, while frozen checkpoints are evaluated on disjoint task-template families to test preference transfer rather than task memorization.Checkpoints occur at 0, 6, 12, 18, and 24 episodes; each checkpoint uses six tasks with three rollouts per task across three seeds.Source: §A.3 · p. 17
  • The primary comparison measures evolving harnesses against no-memory and oracle-context references, with ACE as the main updater and TEPA, TRACE, and Reflexion as alternative recipes under common supervision.The experiment also includes naive full-memory rewrite and corrective-feedback rewrite diagnostics.Source: §A.3 · p. 17
  • ACE proposes incremental add/modify deltas, merges them deterministically, deduplicates similar bullets, and retains at most 12 statements for later episodes.Retained bullets are injected in subsequent episodes; the oldest are removed when the capacity is exceeded.Source: §A.3 · pp. 17–18
  • TEPA replaces keyed precedents, TRACE applies up to 12 learned API checks with at most three attempts, and Reflexion appends up to three reflections without a same-task retry loop.TRACE scores the final attempt even if learned checks still fail; none of these adaptations uses the ground-truth persona checker.Source: §A.3 · p. 18

B.1 PROOF OF THEOREM 4.1

The proofs characterize harness evolution through policy reachability, finite-evidence and capacity bounds, and biased update dynamics. Together, they separate approximation, generalization, and optimization sources of excess risk.

B.1 PROOF OF THEOREM 4.1

  • Context harnesses produce convex mixtures of the frozen model’s concept-conditioned policies, whereas control can transform those policies through action-path intervention.Under the stated risk separation, a user-optimal policy unreachable by context reweighting but reachable by control has strictly lower approximation error in the control class.Source: §B.1 · p. 19

B.2 PROOF OF THEOREM 4.2

  • For fixed interaction count and feedback advantage, the stated generalization bound increases with harness capacity because each memory statement receives fewer observations as capacity grows.The proof allocates ⌊n/L⌋ feedback observations per statement and uses their majority to infer the represented preference.Source: §B.2 · p. 20

B.3 PROOF OF COROLLARY 4.3

  • With d equally likely preferences and at most L memory statements, storing L distinct preferences leaves some preferences unrepresented when L < d, contributing approximation risk.The corollary combines an approximation bound with the finite-interaction generalization result.Source: §B.2 · p. 20, §B.3 · p. 21

B.4 PROOF OF THEOREM 4.4

  • Under contraction, calibration, and persistent update-bias assumptions, nonvanishing displacement of the empirical-risk minimizer under updates yields a positive lower bound on optimization error.The bound relates update displacement and online-output movement to the gap between online risk and empirical-minimizer risk.Source: §B.4 · pp. 22–23

Related papers

Loading 2609.36892v1…