PaperlayerView exact revision on arXiv

Source-linked AI summary

From Question Answering to Task Completion: A Survey on Agent System and Harness Design

Jianyuan Guo, Zhiwei Hao, Chengcheng Wang, Cheng Fan, Tingzhang Luo, Hongguang Li, Ying Gao, Hefei Mei, Jiankun Peng, Rongjian Xu, Minjing Dong, Han Wu, Mengyu Zheng, Kai Han, Shiqi Wang, Chang Xu, Yunhe Wang

arXiv:2606.20683v1cs.AIcs.CL

TL;DR

Long-horizon agents face limits in context, persistent memory, and intrinsic outcome verification, making the runtime surrounding a model an important part of sustained interaction. This survey analyzes agents through a model–harness lens, and concludes that performance is increasingly governed by their interaction rather than model capability alone. It argues that understanding this shift matters for building agents and evaluating progress responsibly.

  • Problem

    LLMs remain vulnerable to finite context windows, weak persistent memory, and limited intrinsic ability to verify long-horizon outcomes.

    Source: §2.3 · p. 6

  • Method

    The survey treats the harness as runtime infrastructure coordinating observations, context, control flow, actions, persistent state, and failure detection and recovery.

    Source: §2.4 · p. 6

  • Results

    The survey concludes that agent performance is increasingly governed by model–runtime interaction rather than model capability in isolation.

    Source: §9 · p. 22

  • Takeaways & Limitations

    Understanding the shift toward full-system agent engineering is essential for building better agents and evaluating progress responsibly.

    Source: §9 · p. 23

    SWE-bench entries differ in model snapshots and inference policies, and vendor rows use closed-source scaffolds.

    Source: §7.3 · p. 17

Abstract

from arXiv · show

LLM-based agents mark a shift from passive question answering to active task completion: they perceive environments, invoke tools, maintain state, and act over extended horizons. As agent systems have evolved from prompt engineering to workflows and context engineering, harness engineering, and agent-native training with co-evolution, a central question has become increasingly important: where does the bottleneck in agent performance reside, in the foundation model, in the execution harness, or in the coupling between them? This survey examines LLM-based agents through a model-harness lens. We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model coupled with an execution harness. We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities: observation, context, control, action, state, and verification. Using this decomposition, we map task properties and domain pressures to harness configurations, review benchmark and evaluation practices, and synthesize model-harness evidence on how runtime design affects long-horizon task completion, efficiency, and reliability. Finally, we identify open challenges in value-aware evaluation, safety, harness generalization, and model-harness co-evolution. Rather than treating agents as models with auxiliary tools, this survey argues that agent quality -- including success, efficiency, safety, and generalization -- emerges from the interaction between model capability, runtime infrastructure, task structure, and evaluation design. A collection of papers discussed in this survey is provided in https://github.com/ggjy/Awesome-Agent-Engineering.

1 INTRODUCTION

The introduction frames LLM-based agents as moving from passive question answering to active task completion and traces an evolution from prompting toward context, harness, and agent-native approaches. It presents an evolution-first, harness-centric synthesis connecting runtime design with task pressures, benchmark evidence, and deployment objectives.

1 INTRODUCTION

  • LLM-based agents move from passive question answering toward systems that perceive environments, reason over goals, and execute multi-step actions.The survey describes modern systems as closed loops that invoke tools, update state, and verify outcomes over extended horizons.Source: §1 · p. 1

1.1 Harness Design as a Performance Lever

  • The survey treats runtime infrastructure as a central performance lever and argues that agent reliability, efficiency, and generalization depend on harness design as well as model capability.It also considers runtimes that compose multiple models and whose routing, orchestration, and verification policies can be optimized.Source: §1.1 · p. 2

1.2 Four Paradigms of Agent Engineering

  • Prompt engineering addresses how to ask, while workflows and context engineering curate what information enters the model and when.Context engineering broadens evaluation from single-answer correctness to whether assembled context enables multi-step task completion.Source: §1.2 · p. 2
  • Harness engineering closes the feedback loop, while agent-native training internalizes agent behaviors and co-evolution updates models and runtimes from execution traces.The four paradigms coexist in practice rather than forming a strict temporal partition.Source: §1.2 · pp. 2–3

1.3 Relation to Prior Surveys

  • Rather than another layer taxonomy or project catalog, the survey traces bottleneck migration and connects harness anatomy with task pressures, benchmarks, and deployment objectives.It distinguishes itself as evolution-first, harness-centric, and attentive to academic evidence and industrial practice.Source: §1.3 · p. 3

1.4 Scope Boundaries

  • The survey covers LLM-based agent systems from 2020 to 2026, prioritizing verifiable sources while treating adjacent non-LLM traditions as complementary rather than surveying them in depth.Its focus is systems where one or more LLMs serve as cognitive engines within an execution harness.Source: §1.4 · p. 4

1.5 Contributions and Survey Structure

  • Its contributions include synthesizing four engineering paradigms, analyzing model–harness pairing, formalizing six runtime responsibilities, mapping task pressures, and motivating value-aware evaluation.The survey also connects these topics to domain adaptations and empirical evidence.Source: §1.5 · p. 4
  • The survey proceeds from agent and harness definitions through scaling limits, engineering paradigms, harness components, task pressures, evaluation, and future directions.It concludes in Section 9.Source: §1.5 · p. 4

2 BACKGROUND AND DEFINITIONS

An agent is defined by its goal-directed closed loop, while an LLM-based agent implements that loop through a foundation model coupled with an execution harness. The harness coordinates runtime responsibilities that shape interaction with the environment and turn model capability into task execution.

2.1 Functional View: What Is an Agent?

  • A functional agent organizes perception, state maintenance, reasoning and decision-making, action, and feedback adaptation around a task objective.Its closed-loop behavior distinguishes agents from single-shot model calls, static retrieval systems, and fixed automation that does not revise behavior as observations change.Source: §2.1 · p. 4

2.2 Implementation View: Model Plus Harness

  • An LLM-based agent couples a foundation model with an execution harness, which supplies the runtime machinery for observations, context, actions, state, and failure handling.The model layer may contain heterogeneous backbone models; in multi-model systems, the harness also decides which model acts at each step.Source: §2.2 · pp. 4–5

2.3 LLM as the Cognitive Engine

  • Models can decompose tasks, adapt through supplied context, and propose structured tool calls, but reliable execution depends on harness validation, dispatch, observation, and repair.Models remain vulnerable to hallucination, finite context windows, weak persistent memory, prompt sensitivity, and limited intrinsic verification of long-horizon outcomes.Source: §2.3 · pp. 5–6

2.4 Harness as the Runtime Substrate

  • The harness is a coordinating runtime substrate for observation, context, control, action, state and artifacts, and verification and governance.Its operational decomposition explains how harness changes can improve performance without changing the underlying model, [23], [24].Source: §2.4 · p. 6

2.5 Key Infrastructure Primitives

  • Infrastructure primitives instantiate harness responsibilities, while conceptual agent operations map many-to-many onto runtime components.For example, perception uses both observation and context, while feedback involves verification, control-loop decisions, and state updates; agent domains stress different components according to their observations, actions, feedback, and safety constraints.Source: §2.5 · pp. 6–7

3 THE LIMITS OF MODEL-CENTRIC SCALING

Model scaling remains important, but its resource costs and increasingly compressed benchmark gains limit model-centric explanations of agent performance. Static, short-horizon evaluations also miss the runtime demands of sustained task completion.

3.1 Resource-Performance Boundary

  • Recent GPT, Claude, and Gemini releases cluster in a narrow high-score range on MMLU-Pro and GPQA Diamond, making later gains less discriminative than earlier generation jumps.This does not mean scaling has stopped working; gains can become smaller and capability-specific while increasing cost, latency, and operational burden.Source: Figure 4 · p. 7, §3.1 · pp. 7–8
  • 7.0M to 30.84M H100 GPU hours accompanied the LLaMA3.1 70B-to-405B increase, yielding gains of 2.6 points on MMLU, 2.6 points on MBPP EvalPlus, and 1.7 points on GSM8K.The 405B model also remained far from near-perfect performance on MATH and MMLU-Pro; Qwen3 gains likewise varied substantially by benchmark.Source: §3.1 · p. 7
  • Because deployed agents make repeated model calls, per-call cost and latency accumulate, and small errors can compound across execution steps.The trade-off between scaling gains and deployment burden is therefore amplified over an agent trajectory.Source: §3.1 · p. 8

3.2 Measurement Boundary

  • When frontier models cluster near the upper range of saturated benchmarks, small score differences become difficult to interpret as differences in real-world agent capability.This measurement boundary motivates evaluating sustained execution alongside model competence.Source: §3.2 · p. 8
  • Static, short-horizon benchmarks evaluate fixed inputs and single outputs, whereas agent tasks require interaction, adaptation, tool use, and recovery from intermediate errors.Evaluation should consider duration, step count, environmental uncertainty, tool-use complexity, state persistence, and recovery demand; one time-horizon study measures task duration at a fixed success probability.Source: §3.2 · p. 8

4 PARADIGM SHIFTS IN AGENT ENGINEERING

Agent engineering has shifted its main effort from eliciting model behavior toward organizing context, stabilizing execution, and training or co-evolving agentic systems. These phases coexist, while each foregrounds a different runtime or model-design challenge.

4.1 Phase 1: Prompt Engineering

  • Prompt engineering elicits and steers existing model capabilities, but does not reliably supply absent knowledge, maintain changing task state, validate actions, or recover from long-horizon failures.Its central problem is how to pose the instruction, prompting the shift toward the information environment surrounding each model call.Source: §4.1 · pp. 8–9

4.2 Phase 2: Workflows and Context Engineering

  • Workflows sequence model calls, retrieval, tools, memory, artifacts, and branching, while context engineering assembles and refreshes those inputs during execution.Context research progresses from external information access through systematic management to explicit context evaluation and adaptive optimization.Source: §4.2 · p. 9

4.3 Phase 3: Harness Engineering

  • Harness engineering addresses execution failures such as lost progress, tool misuse, objective drift, repeated unproductive steps, and failed recovery.The runtime manages observation, action, persistent state, verification, and governance; evidence includes improved coding performance from interface redesign under a fixed model and automatically improved harnesses outperforming hand-designed baselines on Terminal-Bench.Source: §4.3 · pp. 9–10
  • Multi-model harnesses compose specialized models, while learnable harnesses optimize runtime policies such as routing, tool exposure, memory use, and verification.Compositional runtimes and learnable policies support dependable long-horizon task completion through orchestration, verification, and adaptation.Source: §4.3 · p. 10

4.4 Phase 4: Agent-Native Training and Co-Evolution

  • Agent-native training internalizes planning, tool use, verification, and recovery, while co-evolution updates the model, harness, and improvement policy from deployment feedback.Internalization shifts rather than removes the harness: runtime still supplies environment access, state, and safety control; co-evolution governs joint improvement under verification, safety, and cost constraints.Source: §4.4 · pp. 10–11

5 ANATOMY OF THE EXECUTION HARNESS

The execution harness is a coupled runtime of six responsibilities that shapes how model capability becomes task completion. Their configurations interact, so task structure and domain pressures matter alongside the design of each component.

5 ANATOMY OF THE EXECUTION HARNESS

  • The harness decomposes into observation, context, control, action, state, and verification responsibilities rather than independent software modules.Together, these responsibilities determine what the model observes, what enters context, how execution advances, which actions are available, what state persists, and how runs are checked or constrained.Source: §5 · p. 11, §5.7 · p. 13

5.1 Observation Interface

  • Observation interfaces render environment signals for the model, and failures can arise when needed state is absent, noisy, stale, or represented at an unusable level.The central trade-off is richness versus tractability: richer observations improve grounding but increase context cost and distractors, while compression may discard task-critical evidence.Source: §5.1 · p. 11

5.2 Context Manager

  • Context managers select, compress, order, and refresh observations, tool outputs, retrieved evidence, memory, summaries, instructions, and artifacts for each model call.Managed context trades raw-history fidelity for bounded cost; summaries and retrieval can omit or distort state, making faithfulness and integrity central concerns.Source: §5.2 · p. 11

5.3 Control Loop

  • Control loops organize steps, tools, and handoffs, with choices about planning, delegation, coordination, and stopping shaping trajectory stability.Freer loops adapt to unexpected states but risk drift and repeated failure; stronger orchestration improves reliability but can reduce efficiency or overconstrain exploration.Source: §5.3 · p. 12

5.4 Action Interface

  • Action interfaces map model outputs to executable operations and shape tool specification, granularity, routing, permissions, and side-effect control.Low-level tools are general but harder to use robustly, whereas high-level tools improve reliability but may narrow behavior too aggressively.Source: §5.4 · p. 12

5.5 State and Artifact Store

  • State stores preserve progress, traces, plans, checkpoints, diffs, files, and memory across steps, sessions, and subtasks.The trade-off is completeness versus usability: richer state supports continuity and auditability but increases retrieval burden, noise, and stale-memory risk.Source: §5.5 · p. 12

5.6 Verification and Governance

  • Verification checks execution, while governance constrains or repairs it through mechanisms such as approval gates, sandboxing, rollback, retry, and safe termination.Looser constraints permit exploration but raise risks of harmful actions and unrecoverable drift; tighter governance improves safety and recoverability but can slow or block useful behavior.Source: §5.6 · pp. 12–13

5.7 Cross-Layer Interactions in the Harness

  • Harness components interact: richer observations increase context-management burden, expressive actions increase governance demands, and persistent artifacts feed both context and verification.This coupling means task domains and their horizons, oracle strength, and autonomy requirements shape which configuration choices become decisive.Source: §5.7 · p. 13

6 TASK LANDSCAPE AND HARNESS CONFIGURA-

Task structure and domain pressures determine which harness responsibilities become bottlenecks, so harness configuration should follow the task’s pressure profile. Across domains, the priorities shift with horizon, environment, oracle strength, irreversibility, and autonomy.

6 TASK LANDSCAPE AND HARNESS CONFIGURA-

  • Task horizon, environment type, and autonomy define a pressure profile over the six harness components, identifying which runtime responsibility limits performance, reliability, cost, or safety.The central configuration question is which responsibility becomes the bottleneck under the task’s structural pressures.Source: §6 · p. 13, §6.1 · p. 14

6.1 A Harness-Aware Task Taxonomy

  • Long-horizon tasks pressure context, state, and control; open-ended L4 tasks additionally require budget control, stopping criteria, and escalation.The taxonomy characterizes tasks by horizon, environment type, and autonomy level to identify where failures concentrate.Source: §6.1 · p. 13

6.2 Harness Adaptation by Domain

  • Software engineering is verification-dominant, web and GUI interaction grounding-dominant, research synthesis-dominant, medicine safety-dominant, and embodied work control-dominant.Coding’s strong mechanical oracles support generate-test-repair cycles, while weaker oracles in research require provenance and intermediate review.Source: §6.2 · pp. 14–15
  • In embodied settings, layered planning and reactive control connect through persistent skills or state, while irreversible physical actions make verification and governance critical.These tasks span L3–L4, making long-horizon state externalization a baseline requirement.Source: §6.2 · pp. 14–15
  • Domain constraints move the primary bottleneck from verification in coding, through observation and action in web/GUI, to synthesis in research, safety in medicine, and control in embodied settings.Domain labels alone are insufficient; configuration depends on which component absorbs the failure budget.Source: §6.2 · p. 15

6.3 From Task Properties to Harness Configurations

  • Strong automatic oracles favor verifier loops and repair cycles, while weak or delayed oracles shift effort toward provenance, intermediate review, and conservative stopping.Benchmark reports should describe task pressures and harness configurations alongside results, including feedback quality and latency.Source: §6.3 · p. 15
  • Irreversible actions and higher autonomy make verification and governance primary bottlenecks; long-horizon performance across domains relies on externalized state.Context managers and state stores must be designed jointly: summaries determine current visibility, while artifacts and checkpoints preserve recoverability.Source: §6.3 · p. 15

7 EVALUATION AND EMPIRICAL ANALYSIS

Evaluation treats agent scores as outcomes of model–harness pairings and examines how runtime design, task structure, and resource constraints shape task completion. Across coding, terminal, and web benchmarks, results show that both backbone capability and harness choice matter, motivating evaluation beyond success alone.

7.1 Benchmark Landscape and Evaluation Work

  • Benchmarks test distinct harness capabilities, including repository navigation and hidden-test verification, web and GUI grounding, terminal execution, memory persistence, and harmful-action control.Controlled environments, sandboxed execution, standardized rollouts, and trace collection support reproducible, diagnosable comparisons.Source: §7.1 · p. 16, Table 6 · p. 16

7.2 Evaluation Dimensions Beyond Task Success

  • Task success remains primary, but meaningful harness comparisons also consider reliability, efficiency, latency, safety, and process quality.The analysis focuses on task success, runtime, timeout behavior, and token usage when available; public cost fields are sparse and accounting-dependent.Source: §7.2 · p. 16

7.3 Harness Effects on SWE-bench Verified

  • On SWE-bench Verified, GPT-4o ranges from 23.2% with SWE-agent to 38.8% with Agentless, while Claude 3.5 Sonnet ranges from 33.6% with SWE-agent to 53.6% with PatchPilot.Backbone upgrades also matter: SWE-agent + tools rises from 49.0% with Claude 3.5 Sonnet to 73.2% with Opus 4,.Source: §7.3 · p. 17, Table 7 · p. 17

7.4 Harness Effects on Terminal-Bench 2.0

  • Terminal-Bench model upgrades raise scores within fixed harnesses, while harness choices shift the same model’s accuracy by more than 10%.For GPT-5.3-Codex, results range from 64.7% with Terminus 2 to 78.4% with SageAgent; Fig. 6 shows pairings and Fig. 7 summarizes within-model variation.Source: §7.4 · p. 18, Figure 6 · p. 19, Figure 7 · p. 19
  • 14 of 20 Terminal-Bench models vary by at least 10% across harnesses, with a median within-model range of 13.6%.The largest spreads exceed 20%, showing that leaderboard accuracy cannot be attributed to the model alone.Source: §7.4 · p. 18, Figure 7 · p. 19

7.5 Harness Effects on WebArena

  • GPT-4o ranges from 13.1% in the model-only WebArena baseline to 54.6% with Web-Operator, while recent systems exceed 70% with specialized runtimes and strong backbones.Most public reports are complete systems rather than controlled factorial comparisons, so configuration differences remain part of the evidence,, [249].Source: §7.5 · p. 20, Table 9 · p. 20

7.6 Benchmark Insights

  • Harness design should match the benchmark’s feedback structure; scores are conditional on runtime configuration, and future evaluation should include resource use, recovery, safety, and auditability.Compact scaffolds can match richer runtimes when they fit the model’s interaction pattern and the benchmark’s feedback structure.Source: §7.6 · pp. 20–21

8 OUTLOOK AND FUTURE DIRECTIONS

The outlook frames agent quality as value-aware optimization, trainable verification and recovery, and pressure-aware harness design. It emphasizes balancing reusable infrastructure with task-specific adaptations and testing whether improvements transfer.

8.1 From Score to Value-Aware Agent Optimization

  • Value-aware optimization makes task utility explicit alongside cost, latency, risk, reliability, and process quality, rather than treating task success as the sole objective.The trade-off is task-dependent: high-value or high-risk tasks may justify stronger verification, while routine high-frequency tasks favor cheaper models, shorter trajectories, and stricter stopping policies.Source: §8.1 · p. 21
  • Harness engineering is a resource-allocation problem: choices about models, context, tools, verification, retries, and stopping shape progress per unit cost.Future benchmarks should report success with cost, tool calls, retries, P95 latency, recovery behavior, policy violations, and trace auditability.Source: §8.1 · p. 21

8.2 Learning to Verify, Recover, and Adapt

  • Future agents should learn to verify intermediate states, diagnose failures, recover from local errors, and adapt across tasks, not only plan and act.Execution traces can provide outcomes, costs, tool calls, verifier signals, recovery attempts, policy violations, and feedback for learning.Source: §8.2 · pp. 21–22
  • Self-evolution can update the model, harness, or both from execution evidence, but co-evolution raises risks including benchmark overfitting, incorrect failure attribution, stale memory, and unsafe runtime modification.Updates should be retained only after held-out tasks, regression tests, process checks, and safety constraints; high-impact changes warrant human approval.Source: §8.2 · p. 22

8.3 Harness Generalization Versus Specialization

  • Generic harnesses can improve reuse and lower engineering cost but may weaken task-specific inductive bias; specialized harnesses may improve peak performance while reducing transferability and risking benchmark overfit.Evaluation should test transfer across task distributions, model families, and adapters using held-out tasks, cross-domain transfer, component ablations, and runtime profiles.Source: §8.3 · p. 22
  • A layered design pairs reusable infrastructure with domain-specific adapters because harness requirements vary with task pressure profiles.The substrate can provide logging, isolation, permissions, persistence, cost accounting, standardized tool access, and auditability; adapters define observations, actions, verification, memory, and recovery rules.Source: §8.3 · p. 22

9 CONCLUSION

The conclusion presents agent development as a shift across four engineering paradigms, with performance shaped by model-runtime interaction rather than model capability alone. It also notes that reliability and evaluation remain short of realistic deployment needs.

9 CONCLUSION

  • Agent development spans prompt engineering, agentic workflows, harness engineering, and agent-native training, with performance increasingly governed by model-runtime interaction.The harness perspective helps explain why similar base models can behave differently across environments.Source: §9 · p. 22
  • Reliability remains below deployment needs in many realistic settings, and evaluation is only partially aligned with real use.The survey describes the field as still early.Source: §9 · p. 22
  • Agent engineering has moved beyond prompt craft toward studying full systems, a shift the conclusion says is essential for building better agents and evaluating progress responsibly.The boundary between model design and system design is still being renegotiated.Source: §9 · p. 23

Related papers

Loading 2606.20683v1…