PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Aware Harness Provisioning for LLM Agents in Mission-Critical Infrastructure Operations

Liangtao Lin, Qingang Zhang, Zhaomeng Zhu, Tianwei Zhang, Yonggang Wen

arXiv:2608.17433v1cs.AIcs.MA

TL;DR

Existing MCI agents often receive uniform harnesses despite task-specific information and capability requirements, motivating measurable task-to-harness matching. The paper derives task and harness representations, builds literature- and execution-based mappings, and finds that suitable provisioning varies by task and domain rather than having a universal optimum.

  • Problem

    Existing harness strategies often provide the same comprehensive configuration to every task, while prior adaptive approaches generally infer usefulness rather than measure sufficiency.

  • Method

    The paper derives a system-equation-based task taxonomy, defines a cumulative harness hierarchy, constructs two task-to-harness maps, and uses self-check-triggered escalation.

  • Results

    Suitable harness configurations vary across task classes and domains: map-guided escalation improves liquid-cooling accuracy from 0.652 to 0.715, while full provision remains accuracy-optimal for power grids.

  • Takeaways & Limitations

    Harness provisioning follows a domain-dependent accuracy–cost Pareto frontier, so the most comprehensive harness is not universally optimal.

  • Takeaways & Limitations

    The measured task-to-harness relationship requires further validation and possible recalibration for new domains, models, and harness implementations.

Abstract

from arXiv · show

LLM agents have been widely adopted to operate mission-critical infrastructure (MCI). These agents normally rely on a harness that determines what information they can access, which tools they can use, and what actions they can take. Existing systems often expose the same comprehensive harness to every task, which may not be necessary and cause resource wastes. In this paper, we focus on the identification of optimal harness configurations, and view it as a resource-matching problem between what each task requires and what the harness provides. To measure this match, we classify MCI tasks based on the mathematical representation of the underlying system and rank harness configurations by the amount and type of information they provide. We then construct task-to-harness mappings from two sources: mining research literature and measuring controlled agent execution. Leveraging the measured mapping, we propose a new harness provisioning algorithm: map-guided escalation. It begins with a task-specific harness and expands to full provision only after a failed self-check. We evaluate our method in two representative MCI tasks: in liquid cooling, it improves the agent accuracy from 0.652 under full provision to 0.715 and achieves accuracy comparable to Reflexion with 48% fewer tokens; In power grids, full provision remains accuracy-optimal, while map-based provisioning offers lower-cost alternatives. These findings show that harness provisioning follows a domain-dependent accuracy-cost Pareto frontier rather than a universal optimum.

1 Introduction

The paper frames harness provisioning for mission-critical infrastructure agents as a task-to-harness resource-matching problem that balances performance, execution cost, and unnecessary information exposure. It develops comparable task and harness representations, constructs mappings from literature and execution, and proposes map-guided escalation.

  • Problem formulation: The paper formulates harness provisioning as resource matching between task requirements and the information and capabilities a harness provides.This formulation explicitly balances task performance, execution cost, and unnecessary information exposure.
  • Task and harness representation: It introduces a task taxonomy derived from system equations and an ordered harness hierarchy with increasing system access to represent task demand and harness provision comparably.Existing labels such as detection, diagnosis, and prediction lack consistent cross-domain definitions and do not cover the full task space systematically.
  • Task-to-harness mapping: The paper constructs task-to-harness mappings from more than 1,200 MCI O&M papers and controlled execution measurements.The literature analysis finds that future-outcome, latent-state, and intervention tasks tend to use more extensive harness resources.
  • Motivation and results: In liquid cooling, several task categories achieve best performance below full provision while using fewer tokens and shorter time, showing comprehensive harnesses are not always optimal.The introduction motivates task-specific provisioning by noting that full provision can be suboptimal.
  • Provisioning strategy: Using the mapping, the proposed map-guided escalation strategy starts with execution-derived provision and retries with expanded resources when needed.Directly retrieving a corresponding harness level is also described as a simpler mapping-based alternative without an additional routing call.

2 Related Work

Prior MCI operations research spans task-specific pipelines, interactive agent benchmarks, execution-side improvements, and harness-level control of tools, context, safety, and recovery. This work differs by systematically varying harness provision along an ordered ladder while holding the executor fixed to estimate sufficient provision for each task class.

  • Task-Specific MCI Systems: Task-specific MCI pipelines provide strong solutions for detection or diagnosis but do not support a general agent across the operations-and-maintenance lifecycle.Examples include EGADS for anomaly detection, SR-CNN and OmniAnomaly for time-series monitoring [31], and CIRCA for causal root-cause analysis [16].
  • Interactive Agent Benchmarks: Interactive benchmarks evaluate agents in executable infrastructure environments, including cloud incident response, industrial asset maintenance, and traffic-signal control.AIOpsLab covers the incident lifecycle, AssetOpsBench provides industrial operation-and-maintenance scenarios [28], and LLMLight evaluates traffic-signal decisions.
  • Positioning of This Work: Unlike existing benchmarks with predefined environments and interfaces, this benchmark varies harness provision along an ordered ladder, holds the executor fixed, and estimates sufficient provision for each task class.The design directly compares task-to-harness matching rather than changing the executor or environment interface.
  • Execution and Supply-Side Adaptation: Prior agent research mainly improves execution through reasoning, feedback, experience retrieval, planning, tool use, or task-relevant API and instruction exposure.Examples include ReAct, Reflexion [33], ExpeL, AgentGen [8], ToolLLM and AnyTool [4], and Instruction-Tool Retrieval [6].
  • Harness Design and Governance: Recent work treats the harness as a first-class layer spanning execution control, tool access, context, state, verification, recovery, and resource governance.Related systems study portable harness specifications [27], harness search [14], configuration effects [38], programmable tool policies, information-flow constraints [11], sandboxing [40] [42], privilege selection [36], and trajectory boundary compliance [17].

3 Overview

The methodology defines task and harness spaces, estimates their relationship through literature mining and controlled execution, and uses the resulting map to provision harnesses for new tasks.

  • 3 Overview: The workflow has three stages: define task and harness spaces, estimate their relationship, and use the mapping to select harnesses for new tasks.Figure 1 presents this overall methodology.
  • 3 Overview: Tasks are formally defined from the underlying physical system, while harnesses are organized into levels with increasing information and capabilities.This task–harness organization supports comparison between task requirements and harness provisions.
  • 3 Overview: The task-to-harness map combines mined MCI operations-and-maintenance studies with controlled benchmark executions of the same agent under different harness levels in two MCI environments.The literature analysis examines harness levels used for different task classes, while execution empirically estimates the mapping.
  • 3 Overview: For new tasks, direct lookup uses the mapped harness level as final provision, whereas map-guided escalation starts there and retries with the full harness after a failed self-check.Both strategies first assign the task to a class and retrieve its corresponding harness level from the map.

4 Task–Harness Spaces

The section defines task–harness spaces as two comparable rulers: a task taxonomy based on physical-system targets and a cumulative provision hierarchy based on harness access. Together, they enable measurable mappings from task classes to minimum sufficient harnesses while allowing task-specific, nonmonotonic performance effects.

  • Task–Harness Spaces: Two comparable rulers convert harness provisioning into a measurable mapping from task class to minimum sufficient harness.Tasks are organized by what they query or affect in the physical system, while harnesses are organized by the information, tools, and operational access they provide.
  • Task Space: Conventional labels such as detection, diagnosis, and prediction are mapped to coordinates in the common physical-system representation rather than used as primary categories.For example, detection targets current observable signals, diagnosis targets current system mechanisms, and prediction targets future observable signals.
  • Task Space: The task space represents each task by output mode, target time, and target element, yielding 12 canonical classes.The target element distinguishes observable signals, latent states, and system mechanisms; the target time distinguishes current from future outcomes; and the output mode distinguishes reporting from intervention.
  • Harness Space: Harness provision is ordered cumulatively from model-only reasoning through static knowledge and temporal observations to increasingly rich structural and physical access.The hierarchy reflects increasing access to the physical system, not increasing executor intelligence.
  • Harness Space: Higher harness levels do not guarantee better performance because task coordinates do not impose a fixed harness requirement and additional provision can increase cost or introduce irrelevant context.For instance, future-facing tasks may be solved from temporal observations, whereas current-state tasks may require structural models or simulation.

5 Estimating the Task-to-Harness Map

The task-to-harness map assigns each task class a harness level using literature evidence or controlled execution scores. A tolerance-based rule selects the lowest level near the best-supported or best-performing option, balancing sufficiency against unnecessary provision.

  • Mapping rule: The common mapping rule selects the lowest harness level whose support is within source-specific tolerance of the maximum for each task class.This avoids unnecessary provision when several harness levels receive similar support.
  • Literature-derived map: The literature-derived map estimates provisioning practice from harness levels used across more than 1,200 retained studies, with labels assigned by three annotators and consolidated by majority vote.Figure 2 summarizes 1,220 papers across 12 task classes; distributions generally shift toward higher levels for Act, Future, and latent-state tasks, while most classes peak at K3 or K5.
  • Literature-derived map: The literature-derived map captures established provisioning practice rather than verified sufficiency, so frequently used harnesses may remain unnecessary, insufficient, or over-provisioned.It nevertheless provides a literature-scale view of how harness provision varies across task classes.
  • Execution-derived map: The execution-derived map evaluates the same agent under all five harness levels across 240 replayable tasks from liquid-cooling and power-grid environments, then selects the lowest level within tolerance of best measured performance.Only harness provision varies; the agent, reasoning protocol, and evaluation procedure remain fixed.

6 Map-Guided Harness Selection

Map-guided harness selection assigns each task a class-specific initial harness, then escalates directly to full provision when execution fails a self-check. The method evaluates literature- and execution-derived lookup variants alongside conditional Map-ESC escalation.

  • Initial harness selection: The system classifies each task, selects an initial harness from the task-to-harness map, and executes using only its exposed information and capabilities.Selection is a deterministic table lookup after task-class assignment, requiring no additional routing call.
  • Conditional escalation: When the initial self-check fails, Map-ESC expands directly to K5, exposing full information, tools, and operational permissions while reusing findings from the first attempt.A passing initial result is accepted; otherwise the second execution uses the fixed executor under K5.
  • Method variants: The evaluated variants are Lit-Lookup without escalation, Exec-Lookup without escalation, and Map-ESC with execution-map initialization plus self-check-triggered expansion.The lookup variants isolate map quality, whereas Map-ESC tests recovery from instance-level variation.
  • Conditional escalation: Escalation skips intermediate harness levels to avoid repeated execution overhead, while the executor, reasoning protocol, and capability boundaries remain fixed across attempts.Intermediate retries rarely terminate before K5 in the evaluated domains.

7 Experiments

Experiments across liquid-cooling and power-grid environments show that harness requirements vary by task class and domain, making task-aware provisioning useful but not universally accuracy-optimal. Execution-derived maps support efficient direct lookup and effective conditional escalation, with benefits that transfer across executor models.

  • Task and domain effects: The lowest near-best harness level ranges from K2 to K5 in Liqid and K3 to K5 in Grid, with five Liqid and four Grid classes peaking below K5.Prior literature similarly concentrates Inform tasks near K3 and Future, Act, and latent-state tasks near K5, indicating class-dependent requirements.
  • Map estimation and deployment: Exec-Lookup reaches 0.670 on Liqid and 0.762 on Grid while using 0.86× and 0.88× of Full-K5 tokens, whereas Map-ESC raises accuracy to 0.715 and 0.782.Direct lookup favors efficiency, while self-check-triggered escalation improves accuracy at additional token cost.
  • Deployment outcomes: On Liqid, Map-ESC achieves the highest harness-selection accuracy at 0.715, while on Grid Full-K5 remains accuracy-optimal and Exec-Lookup is a lower-cost Pareto-efficient alternative.Task-aware selection improves both accuracy and cost in Liqid, whereas reducing provision in Grid trades accuracy for efficiency.
  • Deployment outcomes: Map-ESC matches Reflexion at 0.715 versus 0.711 while using 48% fewer tokens, whereas ExpeL raises accuracy by 0.016 with 44% more tokens.Map-ESC uses an execution-derived initial harness and at most one fallback to K5, complementing execution-side adaptation.
  • Generalization: The execution-derived map transfers to Qwen3.5-27B without recalibration, improving Liqid accuracy from 0.706 to 0.726 and remaining close on Grid at 0.797 versus 0.808.A map estimated from a single Qwen run performs substantially worse, so reliable estimation benefits from stronger and repeated execution evidence.

8 Conclusion … A.3 Multi-LLM Annotation Protocol

The paper frames harness provisioning as a measurable deployment decision, linking mathematically derived task demands to a cumulative hierarchy of available information and capabilities. Its literature corpus and multi-model annotation protocol provide empirical task–harness mappings, while limitations constrain generalization beyond the evaluated settings.

  • 8 Conclusion: Harness provisioning is formulated as a measurable deployment decision linking a mathematical task taxonomy with a cumulative K1–K5 hierarchy of harness information and capabilities.The task–harness relationship is estimated from both literature-derived provisioning practices and execution-derived mappings.
  • 8 Conclusion: Generalization of the measured task–harness relationship to new domains, models, and implementations requires further validation and possible recalibration.The evaluation covers 240 tasks in two simulation-backed environments and does not fully capture live sensor uncertainty, distribution shift, organizational procedures, or human appr…
  • A Literature Corpus and Annotation: The literature survey covers 13 of 16 CISA critical-infrastructure sectors using sector-only queries, allowing task and harness distributions to emerge from the papers rather than search terms.Defense Industrial Base, Financial Services, and Government Facilities are out of scope.
  • A.1 Corpus Collection and Screening: The screening pipeline reduced roughly 2,000 crawled candidates to 1,223 papers, with 1,220 receiving complete task-coordinate and harness-level annotations.An LLM screening agent discarded surveys, position papers, and papers without a primary MCI operation-and-maintenance task before full-text extraction.
  • A.2 Task and Harness Annotation Schema: Separate task and harness annotation calls prevent label leakage: the task call is method- and name-blind, while the harness call assigns the highest materially used level.The task call returns a method-free primary-task sentence and three taxonomy-axis labels.
  • A.3 Multi-LLM Annotation Protocol: Three independent model providers assign each paper a task coordinate and harness level, with majority voting directly resolving 89% of task-axis labels and 97% of harness labels.Task axes are consolidated separately by majority vote, whereas the harness level is consolidated as a single ordinal label.
  • A.3 Multi-LLM Annotation Protocol: The remaining 30 task-coordinate and 36 harness-level assignments receive evidence-based adjudication without access to original votes, followed by manual verification.Table 4 reports the resulting consensus composition and inter-annotator agreement.
  • A.3 Multi-LLM Annotation Protocol: The project will release an interactive literature explorer containing task coordinates, harness levels, supporting evidence, individual annotations, consolidated labels, and filtering across task and harness spaces.The explorer is intended to support inspection of papers grouped by task class and consensus harness level.

A.4 Corpus-Level Findings … C.4 Domain-Specific Differences

The study combines a literature-derived harness map with controlled benchmarks in liquid cooling and power grids, using cumulative, architecturally enforced harnesses and domain-specific capability boundaries. The corpus reveals systematic provision patterns but is treated as a prior, while benchmark construction uses verified task specifications, controlled splits, and realistic simulation environments.

  • A.4 Corpus-Level Findings: Harness provision rises with task demands: mean level increases 0.45 from Inform to Act, 0.39 from Now to Future, and 3.53, 3.65, and 4.06 across observable, mechanism, and latent targets.K3 and K5 account for 53% and 28% of the literature corpus, respectively; Table 5 summarizes distributions across 12 task classes.
  • A.4 Corpus-Level Findings: The literature-derived map is a prior rather than verified execution evidence because it records prior methods’ harness levels, with sparse classes producing less reliable estimates.It provides a survey-scale account of how MCI operations research provisions system information and capabilities across task classes.
  • B.1 Simulation Environments / B.2 Scenario Generation: The benchmark spans liquid cooling and power grids, with six liquid trajectories and ten Grid2Op day or fault windows covering normal operation, overheating, degradation, overloads, and cascading failure.Liquid models a 10-rack, 130-node/149-edge thermal-hydraulic network; Grid uses IEEE-14 with 14 substations, 20 branches, 6 generators, and 11 loads.
  • B.3 Task Construction: Each domain instantiates 120 parameterized tasks across 12 classes, with deterministic oracle specifications resolved through direct lookups, closed-form physics, or simulator-based evidence.The full benchmark contains 240 tasks and uses approximately 80 oracle methods for Liquid and 54 for Grid.
  • B.4 Construction and Test Splits: Each class splits evenly into construction and held-out test tasks, estimating execution-derived maps only on construction data and evaluating policies only on test data.The full K1–K5 sweep runs on all 240 tasks, with template families stratified across splits under a fixed random seed.
  • B.5 Programmatic and Human Verification: Verification re-fed ground truth to achieve 1.0 on all 240 tasks, audited split leakage and oracle derivations, removed degenerate axes, and re-executed 20 corrected Grid tasks.Human review corrected two Grid ground-truth values before any map or policy results were reported.
  • C.1 Cumulative Harness Manifests / C.2 Tool and Data Interfaces: Harnesses are strictly cumulative from K1 through K5, expose JSON function schemas and structured results, meter simulator time and calls, and cap returned time series and transient rollouts.K5 override conventions cover pump, valve, plate-power, generator/load, and line-outage actions across the two domains.
  • B.6 Representative Tasks / C.3 Capability Enforcement: Capability boundaries are enforced architecturally: agents receive only their level’s registry, while visibility gates clamp observation horizons and mask hidden state at the data source.The grading oracle bypasses these restrictions, and representative tasks are documented in Table 6.

D Execution and Evaluation Protocol … G.2 Per-Class Execution Scores

Across full-benchmark experiments, execution-derived provisioning matches Full-K5 accuracy on Liquid at lower token cost but sacrifices accuracy on Grid, where K5 remains optimal. The evaluation uses deterministic replays, task-specific grading, controlled harness maps, and multiple baselines to characterize this domain-dependent tradeoff.

  • D.1 Executor Configuration / D.5 Repetition and Randomness: Each task uses a frozen gpt-5.4 ReAct-style executor with one call per iteration, a 20-iteration cap, isolated execution, and a 300 s hard timeout with a 45 s earlier soft deadline.Three independent repetitions are run on deterministic replays, with nondeterminism arising only from model sampling; construction and test splits are disjoint.
  • D.2 Task Scoring / D.3 LLM Judge / D.4 Cost Accounting: Scores average deterministic task-specific rule grading with an LLM judge, while token cost includes every executor-side call and latency includes tool and simulator execution.The judge sees only the question, ground truth, and final answer, never harness condition, policy, trace, or experience; judge tokens are excluded from policy cost.
  • E.1 Literature-Derived Map / E.2 Execution-Derived Map: Execution-derived floors select the lowest harness level within εexec=0.05 of the best observed construction-task score, while literature-derived floors use modal levels across all 12 classes.Table 9 evaluates the selected execution-derived floors on disjoint test data after estimating them from construction runs.
  • E.3 Tolerance Sensitivity: The εexec=0.05 operating point lies on a plateau: Grid floors are invariant through 0.05, Liquid changes only two classes, and test accuracy remains flat within noise from 0.05–0.15.Token cost varies by less than 10% across the 0.05–0.15 range, so the tolerance choice does not determine the conclusion.
  • E.4 Execution-derived Level Stability: Adopted floors are re-selected with probability at least 0.7 in 19 of 24 class-domain cells, with instability concentrated among adjacent levels that all solve the class; no resampling falls below K2.Liquid I-F-S illustrates consequentially minor instability because K3, K4, and K5 all solve it.
  • E.5 Statistical Tests: Liquid Exec-Lookup is statistically indistinguishable from Full-K5 (Δ=+0.018, 95% CI [−0.022, +0.061]) at 14% lower token cost, whereas Grid is below Full-K5 (Δ=−0.044) but above Lit-Lookup (Δ=+0.098).On Grid, Full-K5 remains accuracy-optimal; the reported comparisons confirm the execution-derived policy’s lower-cost tradeoff and advantage over literature lookup.
  • F.1 Full-K5 / F.2 Instruction–Tool Retrieval / F.3 LLM-Route and LLM+Exp / F.4 AutoMix / F.5 Blind-ESC / F.6 Reflexion and ExpeL: The comparison includes Full-K5, retrieval, LLM routing with or without experience, AutoMix, Blind-ESC, Reflexion, and ExpeL, while class and task oracles provide non-deployable accuracy ceilings.Per-class full-benchmark means under K1–K5 are reported for all ten tasks in each class, complementing construction-split scores.
  • G.1 Aggregate K1–K5 Sweep: Across all tasks, Liquid K4 can underperform K3 despite improving several classes, while Grid K5 is most accurate and uses fewer tokens than K3 and K4.These aggregate results show that overall averages can obscure per-class behavior and that accuracy-cost ordering differs by domain.

G.3 Construction- and Test-Split Analysis … H.3 Alternative Experience Representations

The analyses validate construction-to-test generalization, identify class-level routing as the effective granularity, and show that direct map-guided escalation outperforms gradual escalation. Absolute costs and alternative experience representations further clarify the method’s domain-dependent trade-offs.

  • G.3 Construction- and Test-Split Analysis: Re-estimating the map on the test split changes the selected harness level in 10 of 24 class–domain pairs, while the construction-split map yields mean test regret of 0.036.The test evaluation uses a disjoint split, and the changed selections primarily occur in less-stable classes.
  • G.4 Absolute Reference Costs: Exec-Lookup averages ≈14.0k tokens/39 s per Liquid task and 11,829 tokens/8.6 s per Grid task after applying the Full-K5 normalization anchors.The anchors are 16,210 tokens and 47.2 s for Liquid, and 13,409 tokens and 7.6 s for Grid.
  • H.1 Routing Granularity and Construction Bias: Class-level lookup reaches 0.670/0.762 accuracy on Liquid/Grid, exceeding family-level routing at 0.634/0.704 and task-level nearest-neighbor routing at 0.639/0.690.Preferred harnesses vary across families within 7 of 12 Liquid and 5 of 12 Grid classes, so class-level floors smooth family-level sampling noise rather than reflecting one template family.
  • H.2 Escalation Diagnostics: Direct K5 escalation achieves 0.715 ± 0.014 on Liquid and 0.782 ± 0.010 on Grid, outperforming gradual escalation at 0.692 ± 0.009 and 0.762 ± 0.010.The gradual alternative has equal Liquid cost but increases Grid cost by 5% tokens and 8% latency.
  • H.3 Alternative Experience Representations: Distilled playbooks help Grid at Full-K5, 0.810 versus 0.806, but not Liquid, 0.648 versus 0.652; routing-only use does not beat free lookup.ExpeL trajectory retrieval is strongest on Liquid at 0.731 but costs 44% more tokens than Map-ESC for a comparable gain.

H.4 Qualitative Error Analysis … I.3 Compute and API Usage

The paper identifies recurring harness-related failure modes and specifies the released artifacts, proprietary-component limits, and compute required for reproducibility. It reports that excessive provisioning can reduce accuracy while increasing execution costs.

  • H.4 Qualitative Error Analysis: Liquid accuracy drops from 0.59 at K3 to 0.46 at K5, while Grid A-N-X falls from 0.69 to 0.65 when simulation enables counterfactual overuse.Sensitivity sweeps can also exhaust the iteration budget, illustrating over-provisioning backfire.
  • H.4 Qualitative Error Analysis: At under-provisioned levels, agents either abstain or extrapolate from typical ranges; Liquid I-N-S at K1/K2 scores 0.00.The passage characterizes this as the honest failure predicted by the taxonomy.
  • I Artifact Availability and Responsible Deployment: The release will include both 120-task corpora, ground-truth specifications and splits, frozen scenario telemetry, harness manifests, tool schemas, literature data, and reproducibility code and records.Execution records include scores, token and latency meters, simulator meters, and tool-call traces; Table 11 reports per-class mean accuracy ± std for every harness level.
  • I.2 Proprietary Components: The liquid-cooling twin remains proprietary, but frozen trajectories, topology metadata, and recorded tool I/O will support reproduction of reported results.Generating new Liquid trajectories requires twin access until the environment is open-sourced.
  • I.2 Proprietary Components: The Grid environment, based on Grid2Op and pandapower, is fully open-source, while the complete Liquid environment is planned for release after open-sourcing.This distinguishes reproducibility of existing Liquid results from generation of new Liquid trajectories.
  • I.3 Compute and API Usage: Approximately 75M tokens were consumed across reported experiments, including 17.1M Liquid and 17.7M Grid executor tokens plus about 2.4M judge tokens.Qwen ran on a single-node vLLM deployment, and simulator computation was negligible at sub-second per solve.
Loading 2608.17433v1…