PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents

Dhaval C. Patel, Kaoutar El Maghraoui, Shuxin Lin, Yusheng Li, Tianjun Feng, Chun-Yi Tsai, Yihan Sun, Wei Alexander Xin, Akshat Bhandari, Tanisha Rathod, Aaron Fan, Sanskruti Vijay Shejwal, Tomas Pasiecznik, Sagar Chethan Kumar, Tanmay Agarwal, Rohith Kanathur, Sam Colman, Amaan Sheikh, Dev Bahl, Ann Li, Krish Veera, Alimurtaza Mustafa Merchant, Shambhawi Baswaraj Bhure, Sajal Kumar Goyla, Chengrui Li, Kirthana Natarajan, Rui Li, Thomas Ajai, Rujing Li, Vivek G. Iyer, Sanjaii Vijayakumar, Yitong Bai, Ayal Yakobe, Darief Maes, Yassine Jebbouri, Tianyang Xu, Thai Quoc On, Vera Mazeeva, Winston Li, Yuval Shemla, Yeshitha Bhuvanesh, Rushin Bhatt, Siddharth Chethan Gowda, Alisha Vinod, Caroline Cahill, Shriya Aishani Rachakonda, Yunfeng Chen, Aryaman Agrawal, Aman Upganlawar, Mao Le Jonathan Ang, Yubin Sally Go, Madhav Rajkondawar, Yang-Jung Chen, Trisha Maturi, Ananya Kapoor, Andrew Li, Shrey Arora, Mana Abbaszadeh, Shen Li, Charles Xu, Byeolah Kwon

arXiv:2606.19704v1cs.AI

TL;DR

Deployed LLM agents are evaluated by aggregate leaderboards that omit important dimensions and may produce unstable rankings across settings. This paper synthesizes benchmark and implementation evidence to propose predictive-validity ranking with a twelve-tier measurement apparatus; existing evidence supports the position only partially and remains insufficient for confirmation.

  • Problem

    Aggregate leaderboards omit deployment-relevant agent dimensions, while public-to-hidden rankings can be unstable across evaluation settings.

  • Method

    The paper synthesizes seven benchmarks and fourteen implementation studies into predictive-validity ranking and a twelve-tier measurement apparatus.

  • Results

    Aggregate-score rankings do not reliably transfer out of distribution, with public–hidden Spearman correlations of ρ = 0.69 for planning and ρ = −0.13 for execution.

  • Takeaways & Limitations

    Agent benchmarks should report multiple deployment-relevant dimensions and prioritize predictive validity over in-sample mean when ranking configurations.

  • Takeaways & Limitations

    The predictive-validity claim has not been tested at scale and is supported by synthesis rather than a controlled randomized trial.

Abstract

from arXiv · show

Agent benchmarks are growing fast, but no single benchmark touches more than four or five of the dimensions that deployment exposes. This paper aggregates the largest coordinated deep-dive of one MCP-based industrial-agent benchmark to date: fourteen parallel implementation studies covering new asset classes (including a multi-modal visual extension), alternative orchestrations, retrieval strategies, reasoning modes, infrastructure optimizations, and evaluation-methodology probes. Consolidating those studies with seven prior agent benchmarks, we argue that aggregate-score leaderboards systematically underspecify deployed-agent evaluation. Rankings derived from aggregate scores do not transfer to out-of-distribution settings; recent public-to-hidden competition retrospectives provide direct empirical evidence of this rank instability. We propose ranking configurations by predictive validity, the correlation between in-sample and out-of-sample rank, rather than in-sample mean, and report a twelve-tier measurement apparatus that exposes the deployment-relevant dimensions HELM and its agent-era successors collapse. The position is operationalized through three falsifiable out-of-distribution criteria with explicit thresholds; existing evidence partly supports it but is too thin to confirm. We close with a pre-registered pilot design and a field-level vision for what the next generation of agentic benchmarks should report.

1 Introduction

The paper argues that aggregate-score leaderboards underspecify deployment-relevant evaluation and should be replaced by multi-tier measurement ranked by predictive validity. It synthesizes seven benchmarks and fourteen industrial-agent implementation studies, while presenting three falsifiable criteria and acknowledging that it runs no new controlled experiments.

  • Motivation: ρ = −0.13 between public and hidden rankings on the execution track of a 149-team agentic competition illustrates rank instability.The reported execution-track sample is n = 13.
  • Central position: Predictive validity—the correlation between in-sample and out-of-sample rank—is proposed as a more useful deployment criterion than in-sample mean.The paper pairs this criterion with a twelve-tier apparatus exposing dimensions collapsed by current leaderboards.
  • Evidence base: The synthesis argues that existing benchmarks measure overlapping subsets of a larger, non-redundant space, while parallel implementations reveal complementary evaluation dimensions.The evidence includes seven benchmark and deployed-system papers plus fourteen implementation studies of MCP-based industrial systems.
  • Contributions: The paper contributes a twelve-tier synthesis, a predictive-validity leaderboard schema, and held-out, cross-subset, and adversarial criteria for falsifiable testing.It also provides implementation guidance and a research agenda that explicitly identifies what remains untested.
  • Scope and limitations: The work is a position paper grounded in synthesis rather than new controlled experiments, framing its evidence as convergent architectural sensitivity rather than independent triangulation.The fourteen studies each modify one architectural variable end-to-end and cover complementary tiers of the proposed apparatus.

2 The Argument

Aggregate-score leaderboards treat qualitatively different agent configurations as equivalent, despite deployment-relevant differences in reasoning, latency, artifact reuse, and retrieval cost. Their validity is further weakened by judge dependence and unstable rankings under distributional shift.

  • Aggregate-score limitations: Aggregate scores can equate reasoning-heavy, retrieval-rich, and tool-hygiene-fragile configurations even though deployment distinguishes their costs, latency, and operational behavior.A Pass1 score of 0.75 may arise from qualitatively different configurations.
  • Aggregate-score limitations: 31 percentage points separate reasoning-on and reasoning-off configurations on clarity-specific scoring despite similar overall rubric means, with data-retrieval and agent-sequence dimensions unchanged.This exposes a per-rubric sensitivity hidden by aggregate means.
  • Aggregate-score limitations: 4.2× separates Plan-Execute and Supervisor-Specialist architectures on turn-2-to-5 latency despite similar single-turn Pass1 scores, because cross-turn artifact reuse is invisible to single-turn benchmarks.The difference reflects multi-turn behavior rather than single-turn performance.
  • Aggregate-score limitations: 50–68% versus ∼90% accuracy and 4.5×–10× token inflation distinguish single-pass RAG from agentic multi-hop retrieval, with neither dominating across deployment constraints.Aggregate scores do not surface this accuracy–cost trade-off.
  • Measurement validity: LLM-as-judge scoring can shift with judge models and prompts, while PHMForge inter-rater reliability reaches only Krippendorff α = 0.61 versus human–human α ∈[0.74, 0.82].Judge-independent measurement is feasible: CAR moves from 0.68 to 0.91 under constrained prompting, a 20pp improvement attributable to prompting rather than backbone-model choice.

3 The Synthesis

The synthesis consolidates seven prior benchmarks and fourteen implementation studies into a roughly twelve-tier measurement apparatus for deployed LLM agents. It argues that current leaderboards report only four or five dimensions, leaving a measurement gap that encourages over-trust in benchmark rankings.

  • Measurement apparatus: The roughly twelve tiers consolidate distinctive contributions from seven source benchmarks and fourteen parallel implementation studies, with each tier intended to represent a non-redundant measurement dimension.The synthesis presents the tier count and orthogonality claim as consolidated from prior work rather than invented.
  • Measurement apparatus: The seven core tiers cover pass-rate floors, tool-call hygiene, planning-process quality, capability axes, cost-efficiency Pareto, failure-mode taxonomies, and reproducibility.These are labeled T1–T7 in the synthesis.
  • Measurement apparatus: The five deployment-extension tiers cover deployment infrastructure, multi-turn dialog, reasoning-mode adaptivity, knowledge augmentation, and evidence.These are labeled T8–T12 and are surfaced by the fourteen implementation studies.
  • Measurement gap: No current leaderboard reports more than four or five of the roughly twelve orthogonal-or-near-orthogonal dimensions of deployed LLM-agent measurement.The synthesis links this coverage gap to systematic over-trusting of benchmark rankings.

4 Predictive Validity as Ranking Criterion

This section proposes predictive validity—correlation between in-sample and out-of-sample rank—as the deployment-relevant ranking criterion, and defines three falsifiable OOD tests with explicit thresholds. Existing evidence partially supports rank instability, but the composite score and remaining criteria require controlled validation.

  • Ranking criterion: Predictive validity should rank configurations by correlation between in-sample and out-of-sample rank, rather than in-sample mean.The rationale is that deployment decisions depend on out-of-sample performance.
  • OOD criteria: The three tests progress from held-out scenarios to cross-subset transfer and adversarial perturbations, increasing distribution shift strength.Adversarial perturbations include paraphrases, identifier renaming, time-window shifts, and distractor injection.
  • Composite score: The proposed composite score combines mean performance with out-of-sample rank reliability, but its weights and functional form remain unspecified pending empirical fitting and validation.Weights would be fit on Criterion-A holdouts to maximize Spearman correlation with Criterion-B/C ranks.
  • Falsifiable conditions: The position is supported if OOD rank correlation falls below 0.85 across at least two criteria, top-3 configurations leave top-5 in ≥10% of holdouts, and PV and mean top-10 overlap has Jaccard < 0.85.A further condition requires mean-vs.-OOD-variance Pearson correlation above 0.2.
  • Current evidence: Public–private execution-track rankings already show Spearman ρ = −0.13, below the 0.85 threshold, while the remaining conditions require controlled study.The authors commit to publishing the position as refuted if the conditions fail under controlled experiment.

5 Convergent Architectural Sensitivity

Fourteen implementation studies reveal strong sensitivity to reasoning mode, retrieval architecture, orchestration, and evaluation anchoring, with aggregate rubric means masking deployment-relevant trade-offs. Across these studies and seven prior benchmarks, all four foregrounded dimensions—robustness, trustworthiness, generalization, and longitudinal/drift behavior—receive convergent evidence.

  • Reasoning sensitivity: 21.5% higher total latency accompanied reasoning-on, while clarity rose 31 percentage points and hallucination fell 7 pp; retrieval and sequence correctness were unchanged.Latency increased from 15.08 s to 18.32 s, and planning latency increased 41.9%.
  • Retrieval sensitivity: Knowledge-Plugin retrieval reached ∼90% rubric accuracy versus 50–68% for single-pass RAG, but required 114–146 s and 4.5×–10× token inflation.With Llama-4-Maverick-17B, RAG ran at 8.9–20 s end-to-end; Granite-3-8B reached 60% at 91 s on the same Knowledge Plugin pipeline.
  • Evaluation trustworthiness: External anchors are necessary for LLM-as-judge scoring: Condition Insight’s Condition Agreement Rate climbed from 0.68 to 0.91 under constrained prompting, while unanchored components cannot detect judge drift.The Condition Agreement Rate gain was attributed to prompting rather than a backbone-model change.
  • Architectural and asset sensitivity: PHMForge replaces the plan-execute base with 99 SME-authored prognostics scenarios across eight asset classes and 39 algorithm-grounded MCP tools, exposing orchestration errors.The benchmark broadens deployment-relevant axes beyond fixed-substrate, fixed-asset, fixed-phrasing evaluation.
  • Convergent evidence: Fourteen implementation teams and seven prior benchmark papers converged on the same measurement dimensions, with robustness, trustworthiness, generalization, and longitudinal/drift behavior each supported by multiple independent teams.The convergence is presented as evidence for the synthesis rather than as a result from a single new experiment.

6 Implications for Leaderboard Design

The paper proposes three concrete leaderboard-design changes: expose configuration axes, present results in predictive-validity and cost-aware layers, and require richer submission artifacts. It also distinguishes the usefulness of aggregate scores for per-model summaries from the non-transferability of aggregate-score rankings.

  • Proposal 1: Declared configuration columns: Leaderboards should declare architecture, reasoning mode, retrieval strategy, prompt-constraint level, and verifier type as separate configuration columns.These non-empty axes affect performance attribution, so conflating them obscures what produced a result.
  • Proposal 2: Layered presentation: Results should use layered presentation: a headline table with PV rank, a cost-Pareto plot, drill-down panels, and significance and confidence intervals.Each layer answers a different question; headline tables should not aggregate beyond their resolution, while details should not bury the headline.
  • Proposal 3: Required submission elements: Submissions should report multi-run variance, hardware disclosure, declared tier coverage, and raw trajectories, alongside shared verification rules and an adversarial-perturbation suite for Criterion C.The proposed community artifacts aim to support judge-independent verification and adversarial evaluation.
  • Alternative views: Aggregate scores remain useful per-model summaries, but rankings derived from them do not transfer; predictive validity instead spans three OOD criteria with explicit thresholds and bootstrap CIs.The paper presents this as a narrower claim about ranking transfer rather than a rejection of aggregate scores for model comparison.

7 Summary and Outlook

The paper synthesizes fourteen implementation studies into a twelve-tier evaluation apparatus and argues that agent benchmarks should rank configurations by predictive validity rather than aggregate mean. It presents field-level recommendations while emphasizing that the position remains a domain-specific, unvalidated synthesis with an unresolved deployment-validity gap.

  • Contributions: Fourteen studies, roughly six thousand judged trajectories, six extension axes, and twelve evaluation tiers expose dimensions that aggregate-score leaderboards systematically underspecify.The studies span asset class, orchestration, knowledge/retrieval, infrastructure, reasoning mode, and evaluation methodology.
  • Recommendations: Four recommendations follow: declare configurations, rank by transfer, require a judge-independent anchor, and adopt persistent non-stdio infrastructure.Configuration fields include architecture, reasoning mode, retrieval strategy, prompt-constraint level, and verifier type; predictive-validity scores should accompany in-sample means.
  • Limitations: Predictive-validity claims remain untested at scale, supported by convergent architectural-sensitivity evidence across fourteen studies rather than a controlled randomized trial.The paper should therefore be evaluated as a synthesis and position, not as an empirical finding.
  • Limitations: Evidence is domain-specific: all studies concern industrial asset operations and maintenance through AssetOpsBench and its extensions, leaving cross-domain generalization open.The twelve-tier apparatus is framed as a hypothesis tested within one domain.
  • Limitations: The twelve tiers are asserted to be roughly orthogonal but have not been empirically tested, and proposed predictive-validity criteria remain internal to AssetOpsBench.No data yet link framework rankings to deployed outcomes such as operator override rate, incident reduction, or false-alarm rate; industry partnerships are required.
  • Evidence and data: The fourteen implementation studies are unpublished reports, valued for convergence under architectural diversity rather than independent peer review.The synthesis uses the publicly released AssetOpsBench dataset under Apache-2.0 and synthetic or publicly available industrial data sources, without human-subjects data or personally identifiable information.

A Additional Convergent-Sensitivity Cases

Five convergent-sensitivity cases show that deployment-relevant performance depends on orchestration, reasoning, routing, asset class, caching, and multimodal serving choices. The reported gains span efficiency, quality, hallucination, and generalization, while also exposing trade-offs and measurement interactions.

  • Orchestration: multi-turn artifact reuse: Supervisor-Specialist artifact reuse reduced tool-time share from 47.3% to 26.3% and raised planning effectiveness from 0.559 to 0.791.Turns 2–5 also ran 4.2× faster than turn 1, at 34.3 s versus 145.4 s mean.
  • Reasoning mode: weight-internalized tool knowledge: QLoRA tool-knowledge internalization raised AT-F1 from 0.47 to 0.65 and judge score from 2.88 to 3.88 while reducing input tokens by 82.6%.The intervention eliminated the approximately 2,400-token schema prepended to every informed-baseline prompt, but the forgetting probe exposed a longitudinal cost.
  • Reasoning mode: confidence-gated routing: At confidence threshold θ = 0.8, routing around Deep-TSFM increased overall-correct from 13.0% to 30.4% and agent-sequence correctness from 6.5% to 88.9%.Baseline audits found unnecessary Deep-TSFM calls in 35.5% of trajectories, with 96% occurring on wrongdomain scenarios; hallucination fell from 93.5% to 35.6%.
  • Asset class: batteries and transformers: Battery and Smart-Grid Transformer extensions both reported 6× to 8× end-to-end speedups under profiling-driven optimization, testing generalization beyond HVAC.The Transformer extension used four IEC-standards-grounded tools, while the Battery analytics server used eleven tools seeded by NASA PCoE cycling data.
  • Knowledge augmentation: caching trustworthiness: The temporal-semantic cache achieved 3.48× end-to-end speedup across 80 paraphrase-tier queries, with a median 30.6× speedup on cache hits.Its hit-decision F1 ceiling was 0.64, with recall 0.5625 and precision 0.75.
  • Asset class (multi-modal): visual inspection extension: In the visual inspection extension, AWQ W4A16 with domain-specific calibration raised Qwen pass rate from 0.48 to 0.82 while reducing latency by 1.99×.The extension covered 22 scenarios across pumps, power transformers, induction motors, and wind-turbine blades, and compared ten serving variants on one L4 GPU.

B Tier Definitions · C Empirical Evidence of Rank Instability · D Per-Study Extended Details

The paper defines a twelve-tier measurement apparatus spanning success, process quality, capabilities, efficiency, failures, reproducibility, infrastructure, interaction, reasoning, knowledge, and evidence verification. Published rank correlations provide mixed evidence of instability, while fourteen implementation studies organize baseline comparisons, improvements, conclusions, and suggestions.

  • B Tier Definitions: The twelve tiers extend evaluation beyond headline success to tool-call hygiene, planning, capabilities, cost, failures, reproducibility, infrastructure, dialogue, reasoning, retrieval, and evidence verification.Each tier names a measurement dimension, source benchmark, and representative metric.
  • B Tier Definitions: Tool-call hygiene captures validity, schema compliance, execution success, and dependency order because hygiene failures can occur on otherwise-successful trajectories.The dimensions are surfaced by MCP-Bench.
  • B Tier Definitions: Planning quality measures judged planning axes, decomposition overlap, Node/Edge F1, and chain-order NED, recognizing that successful configurations may use different planning processes.The cited metrics combine MCP-Bench and TaskBench measures.
  • B Tier Definitions: Capability profiles distinguish similarly ranked models across execution, search, adaptability, time, ambiguity, agent-to-agent interaction, noise, and graph structure.The axes come from ARE/Gaia2, with Node/Chain/DAG stratification from TaskBench.
  • B Tier Definitions: Deployment viability depends on cost-quality Pareto position, failure-mode analysis, reproducibility, infrastructure overhead, multi-turn dynamics, reasoning sensitivity, and retrieval quality.The tiers include $/scenario, latency, variance, recovery rate, context-bloat trade-offs, reasoning attribution, and retrieval recall.
  • B Tier Definitions: Evidence grounding and verification adds judge-independent governance, Unsupported Claim Rate, reasoning stability, and verifier-type declarations beyond LLM-judge-only evaluation.Its representative measures include Condition Agreement Rate and ARE hard, causality, and timing violations.
  • C Empirical Evidence of Rank Instability: ρ = −0.13 with n = 13 and a 95% CI of roughly [−0.64, +0.45] makes the execution-track correlation statistically indistinguishable from zero.The point estimate is informative only as “not robustly positive.”
  • C Empirical Evidence of Rank Instability: ρ = 0.69 with n = 20 and CI ≈[0.35, 0.87] shows robustly positive planning-track correlation, although its upper bound still reaches the falsification threshold.Figure 5 plots point estimates with 95% Fisher z-transform confidence intervals and marks ρ = 0.85.

E Convergence Patterns

Fourteen independent studies converged on five recurring measurement gaps in the benchmark. These patterns expose bottlenecks in execution, tool knowledge, transport, caching safety, and scenario coverage.

  • Pattern A: Plan-execute bottleneck localization: Pattern A localizes a shared plan-execute bottleneck to FMSR or TSFM across four independent groups.TSFM tool-time falls from 47.3% to 26.3% with supervisor routing; FMSR N × M drops from 559s to 15.5s under hedged dispatch; ttm_forward is 93% of single-call latency.
  • Pattern B: Prompt-versus-weight tool knowledge: Pattern B shows prompt-time and weight-time tool knowledge are tunable axes rather than fixed design choices.Approaches include retrieval, confidence-gated routing, adapter-weight internalization eliminating 82.6% of input tokens, and extended-thinking mode.
  • Pattern C: MCP transport overhead: Pattern C identifies MCP transport overhead as a dominant latency floor that the base benchmark conflates with reasoning ability.Residual subprocess overhead is 5–6s per call, discovery costs 2–3s per query, and pre-loading reduces modelloading from 233ms to 26ms.
  • Pattern D: Caching trustworthiness: Pattern D distinguishes caching’s performance benefits from its safety, with parameter collisions producing an F1 ceiling of 0.64.The studies show that “caching helps” and “caching is safe” are separate claims requiring separate validation.
  • Pattern E: Scenario authoring as a binding constraint: Pattern E identifies scenario-corpus size as the binding constraint on benchmark extension work.The original 141-scenario corpus led to alternatives including 15 hand-authored scenarios, a 36-scenario papergrade set, and 99 SME-authored scenarios across eight asset classes.

F Forward-Looking Suggestion Clusters

Fourteen forward-looking suggestions converge into four field-level recommendation families: adaptive routing, persistent MCP infrastructure, cross-model and cross-asset generalization, and scenario-corpus expansion. Together, these clusters identify four axes along which AssetOpsBench remains under-specified.

  • Aggregate thesis: The four clusters converge on the thesis that AssetOpsBench is under-specified in adaptive routing, persistent infrastructure, cross-model/asset coverage, and scenario authoring.The recommendations aggregate suggestions across multiple studies into a single field-level position.
  • Cluster I: Adaptive and learned routing: Five groups converge on adaptive, learned routing because expensive paths are needed only for some queries, motivating a first-class meta-router.Proposals include learned RAG-versus-KP classification, parameter-aware caching, confidence calibration, adaptive ceiling-start dispatch, and reasoning-mode routing.
  • Cluster II: Persistent, non-stdio MCP servers: Three groups identify subprocess spawning and stdio transport as the residual latency floor, recommending persistent, non-stdio MCP servers.Specific proposals include persistent MCP server modes, FAISS-backed cache persistence, and socket reuse.
  • Cluster III: Cross-model and cross-asset generalization: Four groups call for cross-model and cross-asset testing, including additional language models, industrial asset domains, and interchangeable TSFM backbones.Examples span GPT-4, Llama-4, and Mistral Large; pumps, bearings, motors, and AHUs; and asset domains beyond chillers.
  • Cluster IV: Scenario-corpus expansion: Four groups call for expanding the 141-scenario corpus through automated tooling, pending evaluations, continual-learning protocols, and a public living-benchmark plan.G12 reports 25 unjudged scenarios pending.

G Predictive-Validity Score: Methodological Notes … G14 Conf. Gate

The appendix presents predictive-validity scoring as a proposal, with weights fit to maximize cross-criterion rank correlation and a pre-registered pilot that determines whether to revise or proceed with the central claim. The merged study notes also show that agent performance depends on orchestration, caching, infrastructure, reasoning, and verification artifacts rather than aggregate scores alone.

  • G Predictive-Validity Score: Methodological Notes: PV(c) = α ¯Yc − βσYc,OOD−γ IQR(Yc) is proposed rather than finalized, with weights fit on Criterion-A holdouts to maximize Spearman correlation with Criterion-B/C ranks.The proposal constrains α + β + γ = 1 with all weights ≥0 and recommends reporting the full Pareto frontier.
  • G Predictive-Validity Score: Methodological Notes: The pilot decision rule revises the central claim if train–holdout Spearman exceeds 0.95 across all three architectures, proceeds to the full study for correlations in (0.65, 0.95), and treats values below 0.65 as strong support.The supplied passage states that the full study is unlikely to clear the ρ < 0.85 falsification threshold when correlations exceed 0.95.
  • G9 Temporal Cache: Overall performance rose from 13% to 30.4%, hallucinations fell from 93.5% to 35.6%, and sequence performance rose from 6.5% to 88.9% under the optimized reasoning-mode setup.The setup used static plan-execute on four MCP servers.
  • G14 Conf. Gate: Forecasting decreased by 33.3% with unoptimized TSFM (TTM) using HF Trainer, while Chronos was 12.8× faster for forecasting but 81× slower for fine-tuning.The infrastructure comparison demonstrates workflow-specific tradeoffs rather than a universal backend winner.
  • G Predictive-Validity Score: Methodological Notes: Required infrastructure includes a shared rule pipeline for judge-independent CAR verification and an adversarial suite of 30 scenarios with paraphrase, identifier-renaming, time-window-shifting, and distractor-injection perturbations.The submission artifacts are required before any leaderboard can report predictive validity.
  • G9 Temporal Cache: Parameter-rich queries make pure semantic similarity structurally insufficient, motivating parameter-aware caching over entity, sensor, and time-window dimensions.The caching proposal targets retrieval behavior in the G9 Temporal Cache study.
  • G14 Conf. Gate: The strongest agent is described as one that knows when a tool is unnecessary, using a learned confidence calibrator with per-asset thresholds.This framing treats tool avoidance as a reasoning-mode capability rather than universally invoking tools.
Loading 2606.19704v1…