Source-linked AI summary
JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution
Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochen Yu, Zihang Liu, Zhongxiang Sun, Qiankun Li, Yue Liao, Heng Chang, Xiaobin Hu, Qibing Ren, Wangchunshu Zhou, Shuicheng Yan
TL;DR
JIT-Agent addresses manually engineered, task- and instance-dependent harnesses by generating, repairing, and evolving executable harnesses for off-the-shelf agentic LLMs. Across tasks and model families, these harnesses consistently improve backbones, compete with mature runtimes, and can produce gains such as +9.1 on DeepSearchQA for DeepSeek-V4-Flash versus GPT-5.6.
Problem
Agent capability depends jointly on the foundation model and harness, but harness construction remains manual, task-specific, and difficult to scale across heterogeneous task demands.
Method
JIT-Agent generates task-conditioned executable harnesses and trains them through customization, repair supervision, and archive-based evolution under a composable four-module protocol.
Results
+9.1 on DeepSearchQA versus GPT-5.6, while JIT-Agent-generated harnesses consistently improve multiple model families and remain competitive with mature agent runtimes.
Takeaways & Limitations
Harness intelligence is presented as a trainable, transferable capability dimension that complements foundation-model scaling and can improve model–harness cost–performance trade-offs.
Takeaways & Limitations
The four-module harness instantiation is deliberately less rich than production harnesses such as Codex, Claude Code, and DeepSeek Harness.
Abstract
from arXiv · showhide
Agent capability is not determined by the model alone. The agent harness, encompassing memory management, planning strategy, action protocol, and tool/skill orchestration, can dominate the contribution of the underlying foundation model. Yet harness design remains manual, task-specific, and fundamentally unscalable. We present JIT-Agent, a harness intelligence model trained to synthesize task-adaptive agent harnesses on the fly for arbitrary off-the-shelf agentic LLMs. We formalize the agent harness as a composable, machine-generatable artifact governed by a fixed four-module protocol, and train JIT-Agent to customize harnesses for a given task at hand, repair harnesses for stable and reliable execution, and self-evolve by distilling performance signals from an expanding archive of prior harness configurations. Equipped with JIT-Agent as a harness helper, DeepSeek-V4-Flash surpasses GPT-5.6 on DeepSearchQA (+9.1) and OdysseyBench (+4.3), while the already strong GLM-5.2 gains up to +20.2 points. Across controlled evaluations, JIT-Agent-generated harnesses are performance-competitive with mature agent runtimes such as OpenCode and Claude Code and consistently improve multi-scale model families of DeepSeek V4, Mimo-V2.5, and Qwen3.6. To our knowledge, JIT-Agent is the first model purpose-built for just-in-time harness generation, establishing harness intelligence as a trainable, transferable, and compounding dimension of agent capability orthogonal to model scaling.
1 Introduction
JIT-Agent treats harness design as a learnable capability: it generates, repairs, and evolves task-conditioned operational scaffolds around off-the-shelf agentic LLMs. Its four-module protocol and training pipeline improve diverse model backbones and compete with established runtimes.
- Agent capability depends jointly on the foundation model and the harness controlling memory, planning, tools, actions, verification, and recovery.
- AOT harness optimization assumes a durable artifact can generalize across future tasks, domains, or model versions, which is most suitable for stable, homogeneous deployment distributions.
- Different task structures require different harness priors, making the appropriate operational scaffold both domain-dependent and instance-dependent.
- JIT-Agent receives a task, protocol, executable registry, and retrieved prior harnesses, then emits a tailored executable harness that can be revised from feedback while its generator remains fixed.
- Harness intelligence comprises adaptivity, reliability, and evolvability: matching task and backbone, recovering from synthesis failures, and turning feedback into stronger future harnesses.
- JIT-Agent factors harnesses into memory, planning, action, and capability-orchestration modules and trains across customization, repair, and feedback-driven evolution.
- JIT-generated harnesses improve vanilla agents, compete with strong fixed harnesses, and move backbone–harness pairs toward better cost–performance frontiers.
2 Related Work
Related work frames harness construction as optimization over modular executable artifacts, with feedback-driven systems increasingly revising both local instructions and global control structure. Table 1 distinguishes ahead-of-time search, test-time editing, and just-in-time instance synthesis, alongside learned repair and online evolution.
- Harness engineering spans coding, general-purpose, embodied, search, memory-centric, and skill-orchestration systems.
- Explicit interfaces make harness components independently substitutable and optimizable, establishing modularization as a common design language.
- Harness optimization targets executable artifacts, expanding from prompts and declarative pipelines to workflow graphs, role assignments, and complete harness code.
- Table 1 separates ahead-of-time construction, test-time editing, and per-task just-in-time generation, while tracking instance synthesis, learned repair, and online evolution.
- Closed-loop methods use execution traces to diagnose weaknesses, propose edits, and validate which changes persist, including revisions to prompts, memory, tools, middleware, or control flow.
3 Unified Harness Codebase
The paper defines agent harnesses as composable programs built from four interoperable modules under a fixed protocol, enabling diverse existing agent designs to share a common execution framework. HarnessFactory instantiates this design space, while JIT-Agent uses its seed bank and evolving archive to generate and refine task-adaptive harnesses.
- Modularized Harness Design Space: Harness construction becomes assembly over a typed, recombinable design space rather than unconstrained program synthesis.Explicit module dependencies make composability central to just-in-time generation.
- Modularized Harness Design Space: A harness is represented through memory, planning, action, and capability-orchestration modules under a fixed protocol with shared lifecycle, validation, and execution semantics.The protocol preserves principal operational choices while removing incidental language and runtime variation.
- Modularized Harness Design Space: The four-module instantiation is intentionally less rich than production harnesses such as Codex, Claude Code, and DeepSeek Harness.The stated aim is to test whether compact just-in-time harnesses can nevertheless yield substantial gains.
- Modularized Harness Design Space: The runtime maintains immutable event history and mutable controller state, producing tool calls or terminal outputs through a shared execution kernel.Execution begins from an empty history and terminates when a terminal output is emitted.
- Modularized Harness Design Space: Canonical ReAct, engineered ReAct variants, and recursive agents fit the same scaffold through different choices of memory, planning, action, and routing modules.For example, OpenCode uses compact memory and an explicit todo planner, while recursive agents isolate subproblem contexts and route capabilities.
- HarnessFactory: HarnessFactory re-implements 13 representative agentic scaffolds under the shared protocol and kernel, with each complete harness organized by the four conceptual modules.These implementations furnish source material for JIT-Agent’s meta-harness design.
- HarnessFactory: The seed bank anchors initial synthesis, then expands into an archive associating retained harnesses with task, reward, latency, and cost observations.Later archive states provide the incumbent population against which new designs are evaluated.
4 Training Pipeline
JIT-Agent is trained through a lifecycle that synthesizes task-conditioned harnesses, repairs unstable generations, and evolves harnesses using execution feedback and archive states.
- Pipeline overview: JIT-Agent samples harnesses from task context, capability registries, and retrieved prior harnesses, then validates them with a frozen executor.The common utility measure combines task reward, latency, and monetary cost.
- Stage I: Customizing Harness: Stage I combines supervised imitation of accepted teacher generations with preference learning favoring higher reward without efficiency degradation.Preferences require strict improvement in at least one efficiency axis.
- Stage II: Repairing Harness: Stage II converts failed generations and diagnostic reports into short repair trajectories, retaining only harnesses that become executable within two rounds.Diagnostics include compiler errors, interface mismatches, tool-call failures, and runtime exceptions.
- Stage III: Learning to Evolve Harness: Stage III trains Evo-GDPO to propose harnesses that overtake prior designs while preserving reward and improving latency or cost when possible.The method samples candidate groups, normalizes reward and efficiency signals, and applies a PPO-style clipped update.
- Stage III: Learning to Evolve Harness: The evolving harness bank retains candidates only when they match or exceed the current reward frontier and strictly improve reward, latency, or cost.During training, feedback updates both the policy and bank; at deployment, the policy remains frozen.
5 Inference Architecture
JIT-Agent supports static and streaming inference, differing in whether experience is discarded after a task or retained to inform subsequent harness generation.
- Static inference: Static inference generates N harnesses in parallel, selects one, and executes only the selected harness.This increases candidate diversity without increasing environment rollouts.
- Streaming inference: Streaming inference retrieves from an evolving harness bank, executes the selected harness once, and uses feedback to decide whether the experience should be retained.The bank remains unchanged when the completed harness provides no admissible improvement.
6 Experiments and Analysis
Across nine agentic benchmarks and controlled comparisons, JIT-generated harnesses improve matched backbones, compete with advanced fixed runtimes, and often improve both performance and inference efficiency. Gains transfer across model families, tasks, and execution settings, while task-dependent frontiers support generating harnesses rather than selecting one universal scaffold.
- Main Results: JIT-generated harnesses improve all 18 directly matched backbone–benchmark pairs, raising nine-benchmark averages by 7.7 points for GLM-5.2 and 8.8 points for DeepSeek-V4-Flash.The largest gains include 20.2 points on DeepPlanning-Travel for GLM-5.2 and 24.8 points on DeepPlanning-Shopping for DeepSeek-V4-Flash.
- Main Results: JIT-equipped systems achieve the best result in eight of nine benchmark columns, with GLM-5.2 ranking first on seven.JIT-Agent + DeepSeek-V4-Flash leads DeepPlanning-Shopping at 83.9, while DeepPlanning-Travel is the only column not led by a JIT-equipped model.
- Advanced Harnesses: Under fixed-backbone comparisons, JIT-Agent achieves the highest performance in four of six settings and improves several benchmarks over the strongest fixed harness.Examples include +4.7 points on DeepSearchQA and +4.0 on xBench-DS for DeepSeek-V4-Flash, plus +7.0 and +2.9 points on xBench-DS and AgentIF for Qwen3.6-Flash.
- Advanced Harnesses: JIT-Agent uses the fewest tokens and lowest API cost in all six controlled settings, reducing per-case cost by 14.9–54.1% relative to the cheapest fixed harness.The average cost reduction is 36.0%, and some settings combine lower cost with higher performance.
- Cost–Performance Pareto Frontiers: The cost–performance frontier is task-dependent: JIT-Agent can improve performance and reduce cost, or trade some performance for substantially lower compute.On DeepSearchQA, DeepSeek-V4-Flash + JIT-Agent scores 85.1 at $0.066 versus NanoBot’s $0.131, while Qwen3.6-Flash trades 3.9 points for a 51.8% cost reduction.
- Generalization and Evolution: Across 24 matched comparisons, JIT-generated harnesses outperform ReAct by an average of 7.6 points, with gains across DeepSeek V4, Qwen 3.6, and Mimo 2.5 variants.Streaming JIT also finishes above the static variant on DeepPlanning-Shopping, DeepPlanning-Travel, and OfficeBench as feedback accumulates.
- Generalization and Evolution: The qualitative cases show that a shared protocol supports distinct task-specific behaviors, including graph execution with artifact storage and recursive orchestration with a fact store.The protocol constrains interfaces rather than prescribing one execution strategy.
7 Conclusion and Future Work
JIT-Agent turns harness construction into an inference-time capability for synthesizing, repairing, and evolving task-conditioned agent harnesses. The paper frames harness intelligence as a trainable, transferable source of agent capability beyond model weights, while identifying co-design and selective runtime modification as future directions.
- Conclusion: JIT-Agent synthesizes, repairs, and evolves task-conditioned agent harnesses at inference time through a composable four-module protocol.Its training combines customization learning, repair supervision, and Evo-GDPO.
- Conclusion: Across deep research, daily work, planning, and workspace tasks, JIT-generated harnesses consistently strengthen underlying backbones and compete with frontier models and advanced fixed harnesses.The conclusion also reports improved cost–performance trade-offs.
- Conclusion: Harness intelligence is presented as a trainable and transferable source of agent capability beyond model weights alone.The paper defines it as adapting the operational scaffold through which a model acts.
- Future Work: Future systems may combine a stable runtime core with task-dependent construction, revision, or replacement of selected harness components.This is proposed as a less radical production alternative to redesigning the entire scaffold just in time.
- Future Work: The longer-term agenda is model–harness co-design, jointly training foundation models with the operational structures governing memory, planning, action, and tool use.The proposed scaling perspective places harness intelligence alongside model capacity and inference compute.
A Additional Generated Harness Visualizations
The appendix presents task-specific harness visualizations spanning research, travel, phased production, numerical analysis, and workspace execution. Each harness selects state, memory, planning, action, and tool-control mechanisms around the task’s dominant reliability requirements.
- Overview: Eight additional harnesses cover constraint-heavy shopping and travel, phased web production, clue-driven research, evidence-sensitive retrieval, numerical analysis, and workspace manipulation.They are generated under the same four-module protocol but are not variants of one execution template.
- Origami: Origami decomposes a constrained wardrobe request into dependent searches, runs bounded parallel branches, and folds only active working context before aggregation.HierarchicalMemory retains branch trajectories and artifacts.
- Turnstile: Turnstile compiles travel obligations into a typed specification and checklist, tracks evidence buckets, and withholds final_answer until is_complete() succeeds.DynamicToolPolicy exposes searches for missing evidence categories.
- Gearbox: Gearbox uses a shared phase register to switch exposed capabilities and typed memory schemas across search, collection, and construction phases.PhaseAction alone writes the register, while policy and memory modules read it.
- Pegboard: Pegboard represents research as a candidate × clue evidence matrix, using coverage and contradictions to drive targeted search and source-grounded verification.Each extracted observation retains a document identifier.
- Appraiser: Appraiser scores observations by evidence value and renders only the first, last, and top-K items into subsequent context while retaining the full trajectory.The design targets evidence-sensitive reasoning with auditability.
- Abacus: Abacus executes numerical analysis through shell or Python, extracts RESULT_JSON, and writes interpreter-derived quantities into typed state for later steps.This avoids asking the model to re-derive computed values from prose.
- Workspace harnesses: Player Piano performs typed batch file edits with deterministic post-edit checks, while Mulligan retries failed plan steps and commits only successful execution.Their memories preserve task-specific file or execution history at different workspace scales.