Source-linked AI summary
Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
Ruhan Wang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Yue Yu, Junyao Yang, Kishan Panaganti, Haitao Mi, Dongruo Zhou, Leoweiliang
TL;DR
Harness evolution requires locating distributed code sites from behavior-level modification requests, a task that existing implementation-centric repository tools do not fully solve. The paper introduces Harness Handbook and BGPD to organize behavior-linked source knowledge, progressively localize and verify edit sites, and guide planning. Across two open-source harnesses, handbook assistance improves localization and plan quality while reducing planner-token use, with scope boundaries for seed skeletons and unparsed content.
Problem
Modification requests specify desired behavior while harness implementations distribute that behavior across files and stages, making complete behavior localization difficult before editing.
Method
Harness Handbook organizes runtime behavior in source-linked hierarchical representations, while BGPD progressively navigates from relevant stages to verified implementation sites.
Results
Handbook guidance improves plan quality and localization while lowering planning cost: overall win rates rise by 10.0 and 18.9 percentage points and planner token use falls by 12.7% and 8.6% on Codex and Terminus-2, respectively.
Takeaways & Limitations
Making the relationship between behavior and implementation explicit can improve modification planning and localization, while supporting behavior auditing and regression-impact analysis.
Takeaways & Limitations
Function-as-leaf construction assumes a trustworthy seed skeleton and suitable function-level budget, while unparsable or unclassifiable content is conservatively frozen or recorded.
Abstract
from arXiv · showhide
The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and requirements evolve, the harness must be continually modified. Before such a change can be made, a developer or coding agent must identify all code locations that implement the target behavior. This is difficult because production harnesses are large, tightly coupled, and behaviorally distributed, while modification requests describe what the system should do and repositories are organized by files and modules. Code search, repository indexing, and long-context processing ease inspection, but still leave this behavior-to-code mapping to be recovered by hand. Behavior localization is therefore a central bottleneck in harness evolution. We introduce the Harness Handbook, a behavior-centric representation synthesized automatically from a harness codebase via static analysis and LLM-assisted structuring, linking each behavior to its corresponding source. We also introduce Behavior-Guided Progressive Disclosure (BGPD), which guides agents from high-level behaviors to relevant implementation details and verifies candidate locations against the current source. On diverse modification requests from two open-source harnesses, Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens, with the largest gains on scattered sites, rarely executed paths, and cross-module interactions. Evolving complex agentic systems thus depends not only on generating edits, but also on determining where those edits should be made.
1 Introduction
Harness evolution bottlenecks arise because behavior requests must be mapped to distributed implementation sites before complete edits can be planned. Harness Handbook and BGPD make this mapping explicit and improve planning quality and efficiency.
- Modification requests describe desired behavior rather than implementation locations, while production harnesses distribute behavior across files, stages, and shared state.A single behavior may depend on several nonadjacent sites, making complete localization difficult.
- Existing repository representations organize knowledge around files, functions, and modules, leaving agents to infer how scattered code pieces jointly implement requested behavior.This gap persists despite repository maps, code search, summarization, memory, and long-context editing.
- Harness Handbook organizes implementation knowledge by runtime behavior and links each behavior directly to its source code.It is constructed automatically using static program analysis and LLM-assisted behavioral structuring.
- Behavior localization identifies all code locations implementing a requested behavior, making it necessary before a complete edit plan can be prepared.
- Behavior-Guided Progressive Disclosure guides coding agents from high-level behavior descriptions to relevant implementation details in stages.
- Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens on modification requests from two open-source harnesses.
2 Related Work
Prior work treats agent harnesses and repository understanding as important software-engineering concerns, but existing repository representations remain implementation-centric. Harness Handbook differs by explicitly representing operational behavior for localization.
- Harness Handbook differs by introducing an explicit operational behavior representation for behavior localization rather than organizing information only around implementation structures.
- Research increasingly treats the agent harness as a first-class software abstraction extending foundation models into deployable agentic systems.
- Harness engineering research studies constructing, adapting, optimizing, and repairing harness implementations as coding agents become more capable.
- Repository maps, indexing, memory, and natural-language artifacts improve repository navigation, retrieval, and editing by making implementation knowledge more accessible.
3 Harness Handbook
Harness Handbook organizes harness source around runtime behavior in a linked L1–L3 representation, constructs it through static analysis and behavioral organization, and uses BGPD to verify edit sites against the current repository.
- Representation: The Handbook combines an L1–L3 document tree with a state-register view to represent runtime behavior across files, execution stages, and shared states.L1 summarizes the system, L2 describes a selected stage, and L3 links stage behavior to source-grounded implementation entries.
- Representation: Progressive disclosure moves readers from system overview to component overview and unit detail, while behavior–implementation alignment keeps active locators resolvable against the current repository.Invalid locators are frozen and excluded from localization until refreshed.
- Construction: Construction selects a fixed function-as-leaf or file-as-leaf mode to determine L3 granularity and stage-skeleton organization.Function-as-leaf uses a trustworthy seed skeleton when available; file-as-leaf infers the skeleton when no suitable seed exists or function-level organization exceeds budget.
- Construction: The construction pipeline extracts source facts and call edges, organizes source units into execution stages, and synthesizes the linked L1–L3 handbook.Static extraction is deterministic; behavioral organization may use iterative proposal and review for functions or files.
- Modification workflow: The workflow executes the plan and resynchronizes the handbook after every non-empty repository diff, while preserving it unchanged when no diff occurs.Algorithm 1 returns the plan, declarations, updated repository, diff, and handbook package.
- Modification workflow: BGPD localizes requests by selecting relevant stages, tracing shared-state couplings, refining to L3 entries, expanding along call relations, and retrieving source locators.External boundary nodes provide context but are not returned as edit sites.
- Modification workflow: BGPD verifies candidate locators against the current repository before the planner converts source-grounded evidence into edit blocks specifying targets, evidence, and intended changes.The repository remains the authority for implementation details.
4 Experiment
The experiment compares Handbook-Assisted BGPD planning with direct repository exploration on two open-source harnesses, evaluating plan quality, localization accuracy, and planning cost. Results favor handbook guidance across overall quality, reference-plan alignment, request types, and difficulty levels.
- Experiment Setup: The Handbook-Assisted arm uses BGPD-guided source navigation, while the Baseline explores repositories directly; all other requests, models, permissions, and decoding settings are identical.The handbook is built from the same source repository used in the experiment.
- Experiment Setup: The study evaluates 30 behavior-driven modification requests from each of Codex and Terminus-2, covering Query, Cross-file, and Search-Hostile types plus three localization-difficulty levels.Terminus-2 uses function-as-leaf organization, while Codex uses file-as-leaf organization.
- Evaluation: Plan quality is judged on Localization, Scope Control, and Reasoning, with Localization weighted most heavily; comparisons count as wins when scores differ by at least δ = 3 points.Three independent judges—GPT-5.5, Opus 4.8, and DeepSeek-V4-Pro—score plans on a 0–100 scale.
- Better Plans at Lower Token Cost: Handbook-Assisted planning wins overall on both harnesses: 38.3% versus 28.3% on Codex and 45.6% versus 26.7% on Terminus-2.The direction is consistent across all three judges, with Codex gaps of 10.0 points and Terminus-2 gaps ranging from 13.3 to 26.7 points.
- Better Plans at Lower Token Cost: Handbook guidance raises Localization, Scope Control, and Reasoning win rates on both harnesses, with gains of 12.2, 6.7, and 4.5 points on Terminus-2 and 2.2, 1.1, and 3.3 points on Codex.Average planner use also falls from 0.102M to 0.089M tokens per Codex request and from 0.058M to 0.053M per Terminus-2 request.
- Localization and Robustness: All 24 Recall, Precision, and F1 comparisons favor Handbook-Assisted localization, with F1 gains of 5.0–18.8 points across references and granularities.Wrong never increases and falls by as much as 25.9 points, while gains span all request types and difficulty levels.
5 Conclusion
The paper identifies behavior localization as a central challenge in evolving production-scale agent harnesses and presents Harness Handbook as a behavior-centric, source-linked representation supporting guided modification and resynchronization. Experiments show improved planning and localization efficiency without increasing planning-token budgets, while suggesting broader uses beyond modification planning.
- Harness Handbook links behavioral descriptions to distributed source implementations and reorganizes repository knowledge around runtime behavior.It is constructed directly from source and supports behavior-guided progressive disclosure and automatic resynchronization.
- 10.0 and 18.9 percentage points: overall win rates rise on Codex and Terminus-2, while planner token use falls by 12.7% and 8.6%, respectively.These results measure handbook-assisted planning against baseline planning.
- All 24 file- and symbol-level Recall, Precision, and F1 comparisons improve against two independent reference plans, with a weaker planner matching stronger models’ implementation-site localization.The gains persist across request types and localization difficulty levels.
- The behavior-centric, synchronized representation may also support behavior auditing, regression-impact analysis, and future harness self-evolving.The proposed future loop combines localization, planning, execution, and resynchronization.
A Harness Handbook Construction Details
Harness Handbook construction takes repository and configuration inputs through three phases: shared static analysis, mode-specific behavioral organization, and synthesis, validation, and packaging. The pipeline produces a reader-facing handbook with document and register views plus synchronization state.
- The construction algorithm requires repository R, leaf mode g, configuration Θ, and a seed skeleton S0 when function-as-leaf is selected.It returns handbook H containing rendered view V, document tree D, register view Z, fixed leaf mode g, and synchronization state K_g.
- Phase I builds shared static facts, Phase II organizes them behaviorally according to the selected leaf mode, and Phase III synthesizes, validates, archives, and packages the handbook.
- The function-as-leaf branch organizes functions within a seed skeleton, whereas the file-as-leaf branch organizes files and may refine the inferred stage structure.
- Phase III creates the document tree and register view, validates references and source locators, archives construction state, and renders the packaged handbook.
A.1 Phase I: Shared Static Fact Extraction
Phase I performs one deterministic, shared analysis of the repository, using language-specific adapters to build a source-grounded program graph. It records resolved internal and boundary targets while auditing unresolved calls instead of guessing.
- Phase I parses repository R and builds program graph G without LLM involvement.The shared analysis is deterministic across both construction modes.
- Internal-function nodes record qualified names, files, signatures, line ranges, enclosing classes, and observed state access.Boundary nodes represent named external targets such as library functions.
- The graph retains call edges only for resolved internal functions or named boundaries, while unresolved calls are written to an audit log.Later organization therefore builds on the same source-grounded facts.
A.2 Phase II: Behavioral Organization
Phase II organizes static facts around execution behavior using function-as-leaf or file-as-leaf strategies, then Phase III synthesizes hierarchical source-backed descriptions and state relationships. Validation, fallback, caching, and synchronization mechanisms preserve coverage and support later reuse.
- A.2 Phase II: Behavioral Organization: Phase II produces a stage skeleton S and organization state U_g, with function-as-leaf and file-as-leaf differing in their starting points and organization units.
- Function-as-Leaf: Function-as-leaf assigns analyzable functions to stages using source, call-graph, neighboring-assignment, and stage-description context.Functions are normally placed whole but may be divided into contiguous regions serving different behavioral roles.
- Function-as-Leaf: Structural review may add, remove, merge, or split stages, after which affected functions are reclassified against the revised structure.Changes require valid hierarchies and preserved state-register references.
- Function-as-Leaf: Organization rounds stop when the skeleton and assignments stabilize or when the configured budget is reached.A global audit checks whole-function consistency, and retained-region boundaries are separately repaired.
- Function-as-Leaf: Accepted assignments require valid stages, legal source ranges, and nonconflicting actions; parser-aligned boundaries are hashed, while uncertain boundaries receive needs-review status.
- File-as-Leaf: File-as-leaf describes each scanned file, infers an ordered stage skeleton, assigns each file one primary stage, and permits limited secondary stages for cross-cutting files.Missing or unknown assignments remain visible in coverage record Y rather than being forced.
- File-as-Leaf: File-card generation retries failed batches individually or by function-range chunks, and emits fallback cards so scanned files are not silently dropped.Unavailable descriptions are marked undescribed in Y.
- File-as-Leaf: File organization combines call-graph ordering with thematic card-based refinement, using deterministic flat ordering when organization fails.Checks remove duplicates and unknown paths and append omitted files to a fallback group.
C Experimental Details
The appendix details the experimental setup, including the evaluated harnesses, modification requests, evaluation arms, and metrics.
- The setup covers two evaluated harnesses, 30 modification requests per harness, two evaluation arms, and metrics defined in Section 4.1.Requests are balanced across request type and localization difficulty.
C.1 Harnesses and Static Facts
The evaluation uses two production-style harnesses that differ sharply in scale and language, motivating distinct handbook leaf representations. Terminus-2 uses function-level entries, whereas Codex uses file-level entries.
- The evaluation compares Terminus-2, a compact but behaviorally rich Python terminal agent, with Codex, a large Rust coding-agent monorepo.Terminus-2 has six source files; Codex spans thousands of files, multiple surfaces, and deep call graphs.
- Terminus-2 uses function-as-leaf mode, while Codex uses file-as-leaf mode to construct handbook hierarchies at repository scale.The selection reflects the availability of a reliable seed skeleton and the different repository scales.
- Table 2 reports static program-graph facts and the resulting handbook structures for both harnesses.Terminus-2 L3 entries cover functions or contiguous function regions; Codex L3 entries correspond to files.
C.2 Modification Requests
Each harness contributes behavior-driven requests that conceal target locations and vary by request type and localization difficulty. The evaluation includes query, cross-file, and search-hostile changes across three difficulty levels.
- Each harness contributes 30 requests evenly divided among query, cross-file, and search-hostile types.The suites share behavioral intents but phrase them against each harness’s code, producing different implementation sites.
- Query requests modify existing behavior, cross-file requests add capabilities spanning interfaces or pipeline components, and search-hostile requests hide sites in mirrored, fallback, or cold paths.These categories represent distinct localization challenges.
- Requests are labeled Easy, Medium, or Hard according to whether localization requires one behavior, coordination across stages and files, or indirect dependency discovery.Table 4 provides representative requests for each difficulty level.
C.3 Evaluation Arms
The evaluation isolates handbook access by holding the planner, repository, requests, and decoding settings fixed across baseline and Handbook-Assisted arms. The handbook is exposed through a navigable skill and used with BGPD, while its construction combines static facts with LLM-generated behavioral descriptions.
- Evaluation Arms: Both arms use the same NexAU planner, DeepSeek-V4-Pro model, read-only repository tools, requests, repository snapshot, planning contract, and decoding settings.They differ only in handbook availability and corresponding localization instructions.
- Evaluation Arms: The baseline explores files with read-only file reads, in-file search, and directory listing, whereas the Handbook-Assisted arm proposes handbook locators and verifies them against source.The handbook is exposed through a SKILL.md manifest that points to navigable reference files.
- Handbook Construction: Static analysis supplies fixed identities, signatures, source ranges, and call edges, while the model supplies behavioral descriptions and stage organization.The pipeline organizes source units into execution stages and synthesizes an L1–L3 document tree.
- Function-as-Leaf Processing: Function-as-leaf processing assigns functions or contiguous regions to stages, with region boundaries required to be contiguous and statement-complete.Small cohesive functions remain whole; larger functions with multiple phases may be split into two to ten regions.
- Function-as-Leaf Processing: Cross-cutting utilities receive dedicated crosscut stages, while API-surface and subsystem-internal functions follow specialized assignment rules.Logging alone does not make a function cross-cutting.
- Review: A critic reviews each proposed assignment against source behavior, stage definitions, and caller/callee evidence, generally favoring approval to maintain progress.Revision requires a concrete, actionable correctness flaw; rejection is reserved for fundamentally wrong proposals.
- File-as-Leaf Processing: File-as-leaf processing summarizes each scanned file while retaining graph-derived function inventories, line numbers, and call relations as fixed facts.In deep mode, the model reads the whole file and writes the handbook leaf content.
E Illustrative Handbook-Assisted Planning Walkthrough
The walkthrough shows BGPD localizing a behavior-first change request through the Handbook, then verifying candidate locations against live source before planning edits. For triple completion confirmation, this process identifies the gate and its shared state sites, yielding a coordinated counter-based change.
- Change request (Q1: triple completion confirmation): The change request raises the completion requirement from two marks to three consecutive marks before grading.The first two marks re-show the terminal state and ask for confirmation.
- Change request (Q1: triple completion confirmation): Because the request names no file or function, the planner must localize every implementation site for the completion handshake.The walkthrough frames this as a Query-type control-flow revision.
- Phase 1 — Route through the handbook: The Handbook route narrows the behavior to the Completion Gate and links it to the reg-pending-completion register across the loop, initialization, and per-run reset.The route proceeds from the overview and index to the relevant stages, then follows the register’s read and write sites.
- Phase 2 — Verify against the live source: Live-source verification finds seven flag occurrences in one file and resolves the handshake to initialization, per-run reset, and loop read/write sites.The Boolean is set on the first completion mark, tested before the second return, and cleared on non-completion turns and at run start.
- Edit plan: The edit plan replaces the Boolean with a counter that increments on completion marks, resets on non-completion turns, and grades only on the third consecutive mark.The four modify actions cover initialization, per-run reset, and the loop’s two branches, all in terminus_2.py.