Harness Engineering for Software Engineering via Modular Executable Dev-Primitives

1 INTRODUCTION

• Long-horizon agents must repeatedly reconstruct interdependent program state scattered across files, configurations, and tests, a burden history compression cannot remove.Changes to one file can require corresponding changes in callers, configuration, and tests.Source: §1 · p. 1
• The authors frame the approach around whether components can reason about their responsibilities and communicate relevant information directly to one another.The question responds to difficulties in whole-repository migration and other long-horizon software tasks.Source: §1 · p. 2
• Dev-Primitives pair repository artifacts with resident LLMs, enabling localized reasoning, natural-language communication of requirements, and self-modification.The resident model reads the implementation attached to its artifact, keeping component-specific information and reasoning with its owner.Source: §1 · p. 2
• HERMES activates dependency-relevant primitives and uses execution evidence to diagnose implicated components, reactivating only those for revision.The loop continues until the repository passes or the revision budget is exhausted.Source: §1 · p. 2
• 12.4 percentage points on average over matched baseline harnesses; Qwen3-8B Dev-Primitives stay within 4.5 percentage points of homogeneous GPT-5.6 Sol and reduce Terminal-Bench 4.0 inference cost by 26.2%.The comparison uses strong activation and diagnosis models for the Qwen3-8B configuration.Source: §1 · p. 2

Software Engineering Agents and Long-Horizon Evaluation

• Interactive agents inspect repositories, edit code, run commands, and react to runtime feedback, while pipeline methods prescribe localization and repair stages (Yang et al., 2024; Wang et al., 2025; Wang et al., 2024; Xia et al., 2025; Zhang et al., 2024).Other systems distribute work across designer-specified roles or file-level developers under centralized coordination (Chen et al., 2024; Liu et al., 2024; Tao et al., 2024; Wang et al., 2026b).Source: §2 · p. 2
• Evaluation has expanded toward sustained terminal workflows and repository-scale consistency in software evolution, generation, and migration (Merrill et al., 2026; Tang et al., 2026; Ding et al., 2025; Hong et al., 2026).Context growth, semantic drift, and state preservation are identified as bottlenecks for long-running agents (Liu et al., 2026; Wang et al., 2026a).Source: §2 · pp. 2–3

Reusable and Modular Agent Capabilities

• Modular capability work either learns behavior from repository trajectories or explicitly packages reusable, task-agnostic skills, procedures, and tool interfaces (Pan et al., 2024; Yang et al., 2026; Ma et al., 2024; 2026; Jin et al., 2026a;b; Qiu et al., 2025; Li et al., 2026).Dev-Primitives instead attach reasoning to repository-owned artifacts, with requirements flowing along dependency edges and activation and revision driven by task and execution evidence.Source: §2 · p. 3

3.1 OVERVIEW

• HERMES activates relevant Dev-Primitives, modifies their artifacts, executes the repository, and diagnoses failures to reactivate only implicated components.Figure 1 depicts the overall workflow from component selection through diagnosis-driven revision.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 DEV-PRIMITIVES

• A Dev-Primitive wraps an individual repository component with a resident LLM that reasons about and modifies its artifact while communicating requirements to other components.Components can participate by providing information without necessarily changing their own artifacts.Source: §3.2 · p. 4

3.3 HARNESS ENGINEERING VIA MODULAR EXECUTABLE DEV-PRIMITIVES

• Dynamic activation localizes likely behavior-implementing components, expands along dependencies to callers, callees, configurations, and tests, and instantiates only the selected primitives.The cost follows the size of the selected component set rather than the whole repository.Source: §3.3 · p. 4
• Activated primitives exchange directed requirements within a round, allowing dependent components to satisfy cross-file changes without one context holding every modified component.Each primitive reasons over its own artifact and messages addressed to it, then returns an optional artifact modification and outgoing message.Source: §3.3 · p. 4
• The execution environment returns command outputs, exit codes, test outcomes, runtime errors, logs, and stack traces that expose inconsistencies across modified components.Held-out evaluation tests are not used during solving.Source: §3.3 · p. 4
• Bug diagnosis attributes failure to implicated components and supplies revision guidance, after which HERMES can retune objectives, drop components, or activate newly implicated ones.Revision continues until the repository state is accepted or the budget of B rounds is exhausted.Source: §3.3 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation covers four complementary benchmarks and compares HERMES with baselines under matched backbone and reasoning-effort settings; the default revision budget is B = 3.The benchmarks cover issue resolution, whole-repository migration, terminal-based tasks, and DevOps workflows.Source: §4.1 · p. 5

4.2 MAIN RESULTS

• HERMES improves SWE-bench Verified resolved rates for 19 of 20 backbones, ties Claude Opus 5, and reaches 97.0% with GPT-5.6 Sol versus 96.2% with mini-SWE-agent.The resolved rates are reported over 500 instances.Source: Table 1 · p. 5, §4.2 · p. 6
• On SWE Refactor Bench, GPT-5.6 Sol’s composite score rises from 6.5% to 31.0% under medium effort (+24.5), where HERMES helps most.These gains generally require higher inference cost due to iterative execution and revision.Source: Table 2 · p. 6, §4.2 · p. 6
• On Terminal-Bench 4.0, matched HERMES configurations improve resolution by 18.8 and 18.2 points for GPT-5.6 Sol, and 26.3 and 26.7 for GPT-5.6 Terra.Most configurations consume additional tokens and cost, while Claude Sonnet 5 improves resolution while reducing both at either effort level.Source: §4.2 · p. 6, Table 3 · p. 7
• On DevOps-Gym, GPT-5.6 Sol’s average rises from 49.89% with Codex to 55.41% (+5.52 points), with gains across all four categories.The improvement is distributed across build and configuration, monitoring, issue resolving, and test generation.Source: §4.2 · p. 6, Table 4 · p. 7

4.3 SCALING LLMS ACROSS HERMES COMPONENTS

• With Qwen3-8B Dev-Primitives, diagnosis scaling improves Terminal-Bench resolution from 27.6% to 46.1% (+18.5), versus 36.7% (+9.1) for activation scaling alone.Scaling both closes most of the gap to the homogeneous frontier configuration; scaling Dev-Primitives adds 1.2 to 4.5 points across the four benchmarks.Source: Table 6 · p. 8, §4.3 · p. 8

4.4 DEV-PRIMITIVE SELECTION QUALITY

• Selection recall is 95.4% on SWE-bench Verified, 83.3% on SWE Refactor Bench, 84.5% on Terminal-Bench, and 85.7% on DevOps-Gym.The average number of activated primitives ranges from 4.7 to 12.6 per task; solved tasks have 16.5 to 24.7 points higher selection recall than failed tasks, though reference-patch overlap can be conservative.Source: Table 8 · p. 9, §4.4 · p. 9

4.5 ABLATION STUDY

• Replacing Dev-Primitives with a centralized editor over the same selected components costs 14.2 points on SWE Refactor Bench and 13.0 on Terminal-Bench; diagnosis feedback matters most among remaining mechanisms.Inter-primitive communication matters more on SWE Refactor Bench and Terminal-Bench (7.5 and 5.7 points) than on SWE-bench Verified (2.8 points). Increasing the revision budget from 3 to 5 adds at most 0.9 points; removing on-demand activation costs at most 3.0 points and raises inference cost by 1.61–1.88×.Source: Table 9 · p. 9, §4.5 · p. 9

5 CONCLUSION

• Dev-Primitives make repository components active participants, while HERMES instantiates them through dynamic activation, localized collaboration, environment execution, and diagnosis-driven revision.Source: §5 · pp. 9–10
• 12.4 percentage points average gain over matched baseline harnesses across four benchmarks; Qwen3-8B Dev-Primitives stay within 4.5 points of homogeneous GPT-5.6 Sol, with 26.2% lower inference cost on Terminal-Bench 4.0.The latter comparison uses strong activation and diagnosis models.Source: §5 · p. 10

A EXTENDED RELATED WORK

• Prior work covers executable agents, long-horizon and context challenges, and reusable capabilities; Dev-Primitives instead bind reasoning to mutable repository artifacts.The related work contrasts this artifact-bound unit with reusable, task-agnostic behaviors and tools.Source: A EXTENDED RELATED WORK · p. 13
• Unlike centrally coordinated role- or file-level tasks, Dev-Primitives let components reason over current implementations and communicate requirements directly to affected components.This avoids requiring a central agent to reconstruct and relay component-specific state.Source: A EXTENDED RELATED WORK · p. 13

B.1 DYNAMIC PRIMITIVE ACTIVATION

• The activation module analyzes the issue, localizes components and dependencies, decomposes work into objectives, and plans edits; execution evidence can trigger a revised plan or additional activations.It inspects repository structure and task-relevant information as needed rather than placing the entire repository in its context.Source: §B.1 · p. 14, D END-TO-END EXECUTION FLOW · p. 20

B.2 DEV-PRIMITIVES

• Each Dev-Primitive modifies its assigned artifact when needed and communicates task-relevant interface, dependency, configuration, or testing requirements to related activated primitives.Incoming messages reach target primitives before the collaboration round ends, allowing them to revise their local modifications.Source: §B.2 · pp. 14–15

B.3 EXECUTION ENVIRONMENT

• HERMES evaluates the updated repository through actual benchmark-environment execution, collecting shell and build outputs, test results, runtime behavior, and available logs or traces.The environment does not rely on an LLM to predict correctness.Source: §B.3 · p. 15

B.4 BUG DIAGNOSIS; B.5 REVISION AND TERMINATION

• When execution fails, diagnosis returns evidence-linked suspected causes, missed components, and revision guidance to activation; successful edits are retained unless contradicted, and the loop stops at PASS or budget B.Diagnosis cannot override observed task-visible test failures, unsuccessful compilation, or other execution failures; B = 0 still permits the initial plan, collaboration, execution, and diagnosis round.Source: §B.4 · p. 15, §B.5 · pp. 15–16

C.1 ACTIVATION PROMPT

• The activation prompt directs the module to select a compact, task-relevant component set, identify dependencies, assign localized objectives, and revise plans using execution-grounded feedback.It prohibits direct file edits and unrelated activations, and directs the module to preserve prior changes unless new evidence contradicts them.Source: §C.1 · pp. 16–17

C.2 DEV-PRIMITIVE PROMPT; C.3 INTER-PRIMITIVE COMMUNICATION FORMAT

• The Dev-Primitive prompt confines each primitive to its assigned artifact and calls for necessary local changes, preservation of unrelated behavior, and communication of cross-component requirements.The communication format identifies source, target, message type, concise information, and the target's required action; recipients can use messages to revise their modifications.Source: §C.2 · p. 18, §C.3 · p. 19

C.4 DIAGNOSIS PROMPT

• The diagnosis prompt evaluates the repair against the original issue using execution evidence, including tests, regressions, remaining failures, and cross-component consistency, then supplies structured pass or failure feedback.It bars marking a trajectory successful when required tests or benchmark-defined execution checks fail.Source: §C.4 · pp. 19–20

D END-TO-END EXECUTION FLOW

• HERMES activates relevant components, assigns localized objectives, routes their collaboration, executes the modified repository, and uses diagnosis feedback to revise until PASS or the revision budget is exhausted.The maximum budget B counts revision rounds after the initial plan–modify–execute trajectory.Source: §B.5 · p. 16, D END-TO-END EXECUTION FLOW · p. 20
• HERMES applies component changes, executes the repository, and sends shell output, test outcomes, runtime behavior, and traces to diagnosis.Source: D END-TO-END EXECUTION FLOW · p. 21
• When execution fails, diagnosis identifies implicated components and activation revises the affected trajectory, retaining changes unless new evidence contradicts them.Source: D END-TO-END EXECUTION FLOW · p. 21
• B = 0 disables revision; B > 0 permits at most B additional rounds, and observed test, compilation, or other execution failures prevent PASS.Source: D END-TO-END EXECUTION FLOW · p. 21

E BACKBONE MODELS

• Qwen3-8B runs locally through Ollama on NVIDIA A40 GPUs with default sampling settings and a 32K context window; other models use official APIs.Source: E BACKBONE MODELS · p. 21

F.1 EXECUTION ISOLATION AND SETTINGS

• Dynamic activation inspects repository structure and candidate files to select components and assign objectives; bug diagnosis returns a verdict and structured feedback.Each mechanism is implemented by a single LLM call chain or call, respectively.Source: §F.1 · p. 21
• Experiments use seven backbones, with medium reasoning effort and revision budget B = 3 by default unless otherwise specified.Reasoning effort follows the setting reported in each table.Source: Table 10 · p. 22, §F.1 · p. 22
• Task solving can use repository-provided tests and runtime evidence, but benchmark-held-out tests and final grading outcomes are reserved for scoring after HERMES terminates.The same separation applies across SWE-bench Verified, SWE Refactor Bench, Terminal-Bench 4.0, and DevOps-Gym.Source: §F.1 · p. 22

F.2 COMPLETE COMPONENT ABLATION

• Table 11 reports component ablations across three backbones, with the ordering of component importance consistent across GPT-5.6 Sol, Claude Sonnet 5, and Qwen3-8B.Parentheses in the table denote absolute changes relative to complete HERMES with the same backbone.Source: §F.2 · p. 22, Table 11 · p. 23

F.3 REVISION BUDGET

• Increasing B from 0 to 3 adds 12.5 points on SWE Refactor Bench, 12.1 on Terminal-Bench, 9.50 on DevOps-Gym, and 4.8 on SWE-bench Verified; increasing B from 3 to 5 adds 0.5, 0.9, 0.47, and 0.2 points, respectively.Changes are measured relative to the default B = 3.Source: §F.3 · p. 23, Table 12 · p. 23

F.4 DEV-PRIMITIVE SELECTION PROTOCOL

• Selection evaluation compares activated components with benchmark-specific targets and reports micro-averaged recall, precision, and activations per task.Targets derive from developer patches or benchmark supervision; SWE Refactor tasks count as solved when they receive a non-zero composite score.Source: §F.4 · pp. 23–24

F.5 DEV-PRIMITIVE RUNTIME

• Dev-Primitives communicate through directed natural-language messages interleaved with local edits, and the active set can change across revision rounds.Files and other persistent artifacts can be registered as new primitives; transient outputs remain environment observations.Source: §F.5 · p. 24

F.6 ABLATION SETTINGS

• The ablation disables one mechanism at a time while keeping the backbone, reasoning effort, revision budget, and benchmark environment unchanged.Complete HERMES is the reference configuration.Source: §F.6 · p. 24
• Without inter-primitive communication, each primitive edits independently, leaving cross-component information to be recovered indirectly through later planning and execution.Local reasoning and editing remain enabled.Source: §F.6 · p. 24
• Without on-demand activation, every component identified initially stays active throughout subsequent planning and revision rounds.The initial task analysis and component-identification procedure remain unchanged.Source: §F.6 · p. 25
• Without execution feedback, diagnosis relies on static inspection and cannot use test failures, runtime errors, command outputs, or traces to guide later rounds.Activation, primitives, communication, and diagnosis remain enabled.Source: §F.6 · p. 25
• Without diagnosis feedback, raw execution outputs remain available, but the activation module directly interprets them without structured diagnosis or revision guidance.Unlike B = 0, this variant still permits subsequent planning and execution rounds.Source: §F.6 · p. 25

F.7 BENCHMARK-SPECIFIC EVALUATION PROTOCOL

• SWE-bench Verified uses 500 human-validated tasks and reports the percentage of successfully resolved tasks; benchmark evaluation tests are reserved for final scoring.HERMES may run repository-provided tests and reproduction scripts during solving.Source: §F.7 · p. 25
• SWE Refactor Bench evaluates all 20 migrations with a composite score, using its grader only for final evaluation.A non-zero composite score counts as solved in solved-versus-failed analyses.Source: §F.7 · p. 25
• Terminal-Bench 4.0 evaluates 66 tasks across five runs each and reports mean resolution with standard deviation, plus full-trajectory token counts and inference costs.The benchmark grader is used only for final outcomes, not execution feedback.Source: §F.7 · p. 25

F.7 BENCHMARK-SPECIFIC EVALUATION PROTOCOL; F.8 BASELINE PROVENANCE

• DevOps-Gym reports success rates for four task categories and their unweighted average; baseline results combine published or leaderboard values with controlled matched runs where needed.HERMES defaults to medium reasoning effort and revision budget B = 3, and costs include activation, primitive, and diagnosis calls.Source: §F.7 · p. 25, §F.8 · pp. 25–26

Dev-Primitives vs. a Centralized Editor

• 31.0% to 16.8% on SWE Refactor Bench, 51.8% to 38.8% on Terminal-Bench, and 55.41% to 44.21% on DevOps-Gym when replacing Dev-Primitives with a centralized Editor; the reduction is 97.0% to 93.4% on SWE-bench Verified.Both variants use the same activation-selected components; the Editor uses 10–14% fewer tokens and 11–16% lower cost than HERMES.Source: Table 13 · p. 26, G ADDITIONAL ANALYSIS OF DEV-PRIMITIVES · p. 26

Controls for Compute and Context Capacity

• 6.7 points recovered by matching the centralized Editor’s inference budget, but it remains 6.3 points below HERMES at essentially the same token and monetary budget.The control increases candidate revisions per planning round until total usage approximately matches HERMES.Source: G ADDITIONAL ANALYSIS OF DEV-PRIMITIVES · pp. 26–27, Table 14 · p. 27

Efficiency of On-Demand Activation

• 1.61–1.88× higher inference cost and lower task performance result when on-demand activation is disabled; average primitive calls fall from 28.7–64.3 to 9.6–25.1 per task with on-demand activation.The ablation keeps the initially identified components but activates them all throughout the trajectory; primitive-call counts include repeated invocations.Source: G ADDITIONAL ANALYSIS OF DEV-PRIMITIVES · p. 27, Table 15 · p. 27

Run-to-Run Variation on SWE Refactor Bench

• 31.0 ± 1.3 is the three-run composite score for the default GPT-5.6 Sol configuration on SWE Refactor Bench, with individual scores of 29.5%, 32.0%, and 31.5%.The benchmark has only 20 tasks, making its aggregate more sensitive to stochastic variation.Source: G ADDITIONAL ANALYSIS OF DEV-PRIMITIVES · p. 27

Planning and activation

• The activation module selects six components, assigns four to modify and two to inspect, and records dependencies among the selected components.The plan identifies preserving non-ASCII characters across form and serialization paths as the central constraint.Source: H CASE STUDY: A COMPLETE HERMES… · p. 28, Table 16 · p. 28

Collaboration and local modification

• The six activated primitives exchange 15 directed messages, with inspect primitives contributing information while leaving their artifacts unchanged.Messages make cross-component consistency explicit, while four modifying primitives independently add ensure ascii=False across relevant JSON-processing paths.Source: H CASE STUDY: A COMPLETE HERMES… · pp. 28–29

Local modification

• Four of the six activated components are modified in the final diff; the other two remain unchanged after inspection and communication.Source: H CASE STUDY: A COMPLETE HERMES… · p. 29

Execution and diagnosis

• 407 repository-visible regression tests return the same successful exit status as before modification, but three reproduction attempts terminate during Django configuration before reaching the reported behavior.Diagnosis therefore receives no reliable failing-to-passing task-specific execution signal.Source: H CASE STUDY: A COMPLETE HERMES… · p. 29

Held-out outcome

• All 35 held-out tests pass, including the task-specific JSON display test, and the instance is marked resolved=True.HERMES’s patch differs from the developer repair path but satisfies the held-out behavioral evaluation.Source: H CASE STUDY: A COMPLETE HERMES… · p. 29

Metric interpretation

• Reference-overlap metrics measure agreement with the developer repair path, not every component set capable of supporting a correct repair.The case illustrates that an alternative behaviorally valid path may receive low reference overlap.Source: H CASE STUDY: A COMPLETE HERMES… · p. 29

• HERMES adds inference cost and latency through multiple primitive calls, repository execution, and iterative diagnosis; gains are smaller in saturated settings such as SWE-bench Verified with the strongest backbones.Source: I LIMITATIONS · p. 29
• Missed dependencies or sparse runtime signals can limit recovery because performance depends on activating relevant components and obtaining informative execution feedback for revision.Source: I LIMITATIONS · p. 29
• The study uses files as the default Dev-Primitive granularity and evaluates four software engineering benchmarks, leaving finer-grained primitives and broader settings for future work.Qwen3-8B is served locally with Ollama on NVIDIA A40 GPUs, while other models use official APIs; results may vary with deployment settings and model updates.Source: I LIMITATIONS · pp. 29–30