PaperlayerView exact revision on arXiv

Source-linked AI summary

When Are Aggregate Agent Traces Diagnosable? Traffic-Governed Interpretation and Calibrated Abstention

Peiying Zhu, Sidi Chang

arXiv:2609.25806v1cs.AI

TL;DR

Closed-loop policies shape which states runtime traces expose, so aggregate behavior may not show a fault in rarely visited regions. The paper gates fault interpretation on reference and matched-runtime support, then analyzes stable signals using affected clean traffic. In the heldout, the gates admitted 55/72 reference units and 54/55 matched runtime units; affected clean traffic outperformed cell coverage, while the study did not confirm localization accuracy.

  • Problem

    A component fault may leave no visible aggregate trace when the policy rarely visits the affected region, while demand shift can alter visitation and action summaries without a fault.

    Source: §1 · p. 1

  • Method

    The workflow requires repeated clean-reference support and joint clean/current runtime support before stable-signal analysis, and models exposure with affected clean traffic.

    Source: §1 · pp. 1–2

  • Results

    55/72 reference units were admitted and 54/55 passed matched runtime admission; affected clean traffic outperformed cell coverage, and stable false admission was 0/20 (one-sided 95% upper 0.1391).

    Source: §10 · p. 6

  • Takeaways & Limitations

    Aggregate traces should be interpreted only after establishing component exposure, with abstention when the trace cannot support the claim.

    Source: §10 · p. 6

    The heldout confirms pre-localization signal behavior, not planted-fault localization accuracy.

    Source: §1 · p. 2

Abstract

from arXiv · show

Runtime traces can appear transparent, but a closed-loop policy determines which states are visited and which failures become visible. We study a simulated hotel-pricing agent mapping time, inventory, and market state to discrete price actions under varying demand regimes. A fault may leave no aggregate trace when the policy rarely visits affected cells. We treat entry into aggregate-only fault interpretation as a diagnosability decision preceding scoring or localization. A reference-map gate requires repeated clean-policy support; a matched runtime gate then requires joint support in clean and current streams. Signal analysis occurs only after both pass. We calibrate false admission on a disjoint clean stream at the physical-component level and model detection by affected clean traffic rather than nominal cell coverage. In a frozen one-shot heldout, 55/72 (76.4%) regime-component units were reference-admitted, representing 20 physical components; 54/55 passed matched runtime admission, while the rejected unit abstained. Stable false admission was 0/20, with a one-sided exact 95% upper bound of 0.1391, meeting the frozen 0.20 criterion. Across 540 repeated unit-arm rows nested in those 20 clusters, affected clean traffic reduced negative log likelihood by 29.3% relative to cell coverage, a gain of 0.1264 nats per row (cluster-bootstrap 95% interval [0.0593, 0.1918]). Adding mask family and its interaction improved log loss by 0.0015 nats per row (one-sided upper bound 0.0066), below the frozen 0.01 practical-sufficiency margin. A development audit found that exact minimum hitting set and greedy selection chose identical supports in 12/12 scenarios because singleton evidence had resolved the conflicts. The result is a bounded rule for interpreting aggregate agent behavior: first establish exposure, then score change, and abstain when the trace cannot support the claim.

1 Introduction

Closed-loop policies shape which states appear in runtime traces, so a component fault may leave no aggregate change when affected regions are rarely visited. The paper places support-based admission before signal analysis and localization.

1 Introduction

  • A component can fail without a visible aggregate change when the policy rarely visits its affected region, while demand shift can change visitation and action summaries without a fault.The paper argues that interpretation which ignores exposure can mistake missing evidence for reassurance or ordinary drift for a fault.Source: §1 · p. 1
  • Exact MHS and propagation-aware greedy selection chose identical supports in all 12 development scenarios because planted-component singletons had already resolved conflicts.The audit redirects the paper’s focus upstream: whether a trace supports a component-level interpretation.Source: §1 · pp. 1–2
  • The workflow requires repeated clean-policy support at a reference-map gate and joint clean/current support at a matched runtime gate before stable signal is evaluated.Clean-versus-clean false admission is calibrated separately, with physical components as the safety sampling unit.Source: §1 · p. 1
  • Affected clean traffic measures the fraction of clean visits in changed cells, addressing why the number of edited cells need not predict signal strength.The paper compares this behavior-grounded exposure measure with affected cell count across uniform and flow-weighted masks.Source: §1 · pp. 1–2
  • The frozen shift detector triggers map recomputation rather than declaring a fault or an abstention, separating distribution change from support loss and stable fault signal.The heldout confirms gates, abstention, and traffic-indexed signal behavior, but not localization accuracy.Source: §1 · p. 2

2 Related work and conceptual position

Prior work provides trajectory benchmarks, diagnosis methods, and support-aware inference, but these do not by themselves establish when aggregate closed-loop traces support a component-level fault interpretation.

2 Related work and conceptual position

  • Interactive-agent benchmarks retain trajectories and environment state, but those records do not automatically establish that a trace statistic identifies a component-level failure.This work asks whether an aggregate trajectory is eligible to support fault interpretation in the first place.Source: §2 · p. 2
  • Conflict-directed diagnosis and exact hitting-set solvers operate on compiled conflicts, whereas diagnosability asks whether available measurements distinguish faults.The proposed gates are a finite-sample operationalization for aggregate closed-loop traces, not a general diagnosability theorem.Source: §2 · p. 2
  • The paper’s empirical contribution is a pre-localization pipeline combining support gates, component-level false-admission calibration, and traffic-indexed signal analysis.It does not claim to originate diagnosability, abstention, overlap analysis, shift detection, or behavioral diagnostics.Source: §2 · p. 2
  • The paper applies support-first reasoning before fault interpretation, with fault visibility—not treatment effect or policy value—as the estimand.It also keeps drift-to-refresh separate from fault interpretation and complements runtime diagnostics and agent-behavior tools.Source: §2 · p. 2

3 Problem setting

The problem setting uses aggregate traces from a simulated pricing agent and treats whether those traces support interpretation as an explicit, staged decision.

3.1 Components, regimes, and aggregate traces

  • The study forms 72 regime-component units from 24 disjoint target-policy components evaluated under three separately fitted demand regimes.Each component is indexed by a time quarter, inventory half, and market third; typed aggregate traces supply the observed behavior, not planted fault identity.Source: §3.1 · p. 2

3.2 Fault masks and directions

  • Formal faults change deterministic subsets of direction-changeable target-field cells by one action bucket, tested in outward and inward directions.The uniform and flow_weighted mask families can select the same cell fraction yet affect different amounts of clean traffic.Source: §3.2 · p. 2

3.3 Affected clean traffic

  • Affected clean traffic measures the fraction of clean reference visits in changed cells, while the comparison covariate uses the mean outward and inward selected-cell fractions.Zero-occupancy components have zero affected traffic and are reported separately rather than shifted by an arbitrary constant before logging.Source: §3.3 · p. 3

3.4 Diagnosability as a state, not a hidden assumption

  • A unit that fails reference support never reaches the runtime gate, and one that fails matched runtime support never reaches signal interpretation.This distinguishes having no usable comparison from having a usable comparison with no stable signal.Source: §3.4 · p. 3

4 Why MHS is downstream

The development audit found no optimizer advantage for exact MHS over propagation-aware greedy selection in this benchmark. The evidence construction had already resolved the conflicts before solver choice.

4 Why MHS is downstream

  • Exact MHS and propagation-aware greedy selection chose identical supports and achieved identical exact recovery (9/12) in 12 audited development scenarios.Non-singleton conflicts were 55.5% of the collection and conflicts overlapped in all 12 scenarios, yet planted-component singleton conflicts appeared in all exact-anchor and hard-probe cases.Source: §4 · p. 3
  • Exact-anchor predicates paired with one-component probes produced planted-component singleton conflicts, so propagation forced the choice before exact and greedy solvers could differ.The authors retain MHS as an optional downstream risk-cost optimizer and make no formal localization claim.Source: §4 · p. 3

5 Diagnosability method

The method gates aggregate-trace interpretation on repeated clean reference support and joint clean-current runtime support, then evaluates stable signal. It reports coverage loss, stable false admission, and drift refresh as separate outcomes.

5.1 Reference-map gate

  • A unit enters the reference map only when at least 14 of 15 clean partitions have support, with each partition requiring a support count of at least 12.The admitted proportion over all 72 units is the primary coverage description; represented physical components are also reported.Source: §5.1 · p. 3

5.2 Runtime two-stream gate

  • Runtime admission requires at least 14 of 15 matched partitions with reference and current support both at least 12; both directions must pass for a bi-directional observation.Separate totals can conceal mismatched partitions: the supported reference and current partitions must coincide.Source: §5.2 · p. 3

5.3 Stable aggregate signal

  • After runtime admission, a direction is stable when at least 14 of 15 matched pairs signal at z >= 1.50, and a bi-directional unit-arm requires stability in both directions.The 1.50 threshold was selected before formal execution from a fixed grid and was not tuned on the formal null.Source: §5.3 · p. 3

5.4 Three separate failure quantities

  • Reference-map invalidation, runtime two-stream rejection, and stable false admission are reported separately because they represent distinct failure quantities.Only stable false admission has a frozen population-rate threshold; the other two are descriptive coverage endpoints, and affected units abstain.Source: §5.4 · p. 4

5.5 Drift and refresh

  • A frozen split-conformal detector requests atomic map recomputation on an alarm, but the formal smoke test does not show that future data can repair a stale map.Refresh reuses the same frozen reference buffer; passing units return to MAP_VALID and units failing the refreshed reference rule enter REFERENCE_ABSTAIN.Source: §5.5 · p. 4

6 Frozen confirmatory design

The confirmatory design separates clean reference, current/fault, and unused seeds, and treats physical components—not simulator episodes—as inference clusters. It seals traffic-based mask construction before fault outcomes are opened and specifies the formal endpoints.

6.1 Seed separation

  • The design uses 15 clean reference partitions and 15 disjoint current/fault partitions, while 200 unused seeds remain reserved.Common random numbers pair conditions, and repeated observations are clustered by physical component; simulator episodes are not independent statistical units.Source: §6.1 · p. 4

6.2 Formal mask construction

  • Deterministic nested cell-order prefixes target affected clean traffic near 0.15, 0.30, 0.50, 0.75, and 0.95, with achieved traffic entering analysis.The clean stream determines occupancy and masks before fault outcomes are opened, and selected-cell, traffic, cell-fraction, and policy hashes are sealed.Source: §6.2 · p. 4

6.3 Formal endpoints

  • Formal endpoints cover admission, abstention-related coverage, stable false admission, signal power versus achieved traffic, traffic-versus-cell log loss, family sufficiency, monotonicity, and state-machine execution.MHS output and planted localization accuracy are excluded.Source: §6.3 · p. 4

6.5 Development separation

Development data informed the choice of traffic as the primary analysis axis but were kept separate from the formal heldout.

6.5 Development separation

  • Development data motivated traffic as the primary axis but were not pooled with the formal heldout or used to establish family sufficiency.The 20-to-15-partition projection and withdrawn cell-coverage heuristic were development-only, not formal endpoints.Source: §6.5 · p. 5

7 Formal results

The frozen heldout admitted 55 of 72 regime-component units, with one runtime rejection, and observed no stable clean-versus-clean false admission among runtime-admitted components. Traffic outperformed cell count for modeling signal, while family sufficiency and drift-refresh findings remain bounded by their stated criteria and design.

7.1 Sampling frame, admission, and abstention

  • 55/72 regime-component units passed formal reference admission, representing 20/24 components; one of 20 component clusters was rejected at runtime, and affected units abstained.Reference invalidation was 0/20 at component grain and 0/55 at regime-unit grain; these coverage rates have no population pass threshold.Source: §7.1 · p. 5
  • 0/20 runtime-admitted components produced a stable clean-versus-clean signal; the one-sided exact 95% upper bound of 0.1391 met the frozen <=0.20 safety rule.The corresponding regime-unit result was 0/54 and remains descriptive.Source: §7.1 · p. 5

7.2 Traffic-indexed power

  • All 44 zero-exposure scheduled observations abstained and none was stable; the conditional denominator was 54 runtime-admitted units, versus 55 reference-admitted units operationally.The operational denominator counts the one runtime abstention as no actionable diagnosis.Source: §7.2 · p. 5

7.3 Traffic versus cell coverage and family sufficiency

  • 29.3% lower negative log likelihood for the traffic model than the cell-count model was observed across 540 unit-arm rows, a difference of 0.1264 nats per row.The component-bootstrap 95% interval for the difference was [0.0593, 0.1918].Source: §7.3 · p. 5
  • 0.0015 nats per unit-arm row was the family-plus-interaction model's log-loss improvement; its one-sided component-bootstrap 95% upper bound of 0.0066 was below the frozen 0.01 margin.The sufficiency label is limited to the two frozen mask families and the precommitted criterion; the result does not exclude family gaps on the odds scale.Source: §7.3 · p. 5

7.4 Drift, refresh, and decision summary

  • The drift detector produced 0/15 alarms in the matched lambda0=7 regime, versus 15/15 at lambda0=5 and 14/15 at lambda0=9 under demand shifts.These are descriptive state-machine inputs, not a false-alarm estimate; refresh reused the same buffer and did not test repair of a stale map.Source: §7.4 · p. 5
  • Execution, operational safety, descriptive coverage, and the three structural hypotheses retain separate labels and interpretations rather than a combined scientific verdict.The passage does not combine these findings into a single overall judgment.Source: §7.4 · p. 5

8 Discussion

The endpoint pattern bounds the claims to admitted units and the two frozen mask families, while abstention marks cases without actionable evidence. The discussion frames exposure and matched support as prerequisites for interpreting aggregate traces.

8.1 Interpretation is fixed by the endpoint pattern

  • Stable false admission supports only a component-cluster 0.20 upper-bound statement; invalidation and runtime rejection are descriptive coverage losses with abstention.Nondecreasing counts are a weak shape check, not evidence of family equivalence.Source: §8.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6

8.2 What the work changes conceptually

  • Exact MHS can optimize a compiled objective without identifying a physical fault, especially when singleton evidence already resolves the conflicts.The workflow places reference admission, matched runtime admission, and stable-signal assessment before localization; traffic exposure also explains why maps can move with environment.Source: §8.2 · p. 6

8.3 Why abstention is part of the result

  • Admission below 100% defines the supported component universe, so operational power counts abstentions as no action alongside conditional power.Reporting both distinguishes a narrow high-performing method from a broader moderate one.Source: §8.3 · p. 6

9 Limitations and claim boundaries

The formal study’s claims are bounded by its simulator, shift setting, available component clusters, and calibration assumptions.

9 Limitations and claim boundaries

  • The formal study uses one simulator and one-bucket target-field shifts, so other shift magnitudes, interactions, persistence patterns, and non-target-field faults may yield different curves.The study has only 24 physical clusters; its exact-binomial statement remains conditional on an exchangeable-independent component working model despite common seeds.Source: §9 · p. 6
  • The false-admission bound is 0.20 rather than 0.05, and the study confirms pre-localization signal behavior rather than planted-fault localization.The bound is conditional on the exchangeable-independent component working model despite common seeds.Source: §9 · p. 6
  • Practical sufficiency applies only to two frozen mask families and a 0.01-nat margin; transferring simulator-specific gates or thresholds to deployed agents risks false assurance without new calibration.Percentile cluster-bootstrap intervals from about 20 represented clusters may undercover in directions favorable to both structural findings.Source: §9 · p. 6

10 Conclusion

The conclusion presents aggregate-trace interpretation as bounded by exposure and admission, with ambiguous traces remaining unresolved. Figure 1 relates stable-signal probability to achieved affected clean traffic across two mask families.

10 Conclusion

  • 55/72 reference units were admitted and 54/55 passed matched runtime admission; stable false admission was 0/20 with a one-sided 95% upper bound of 0.1391.Affected traffic outperformed cell coverage and met the 0.01-nat sufficiency criterion within the two frozen families.Source: §10 · p. 6, Table 4 · p. 8
  • Exact MHS did not improve localization over propagation-aware greedy selection, while behavior under genuinely ambiguous traces remains open.The conclusion’s bounded rule is to establish exposure before scoring change and abstain when the trace cannot support the claim.Source: §10 · pp. 6–7
  • Figure 1 plots bi-directional stable-signal probability against achieved affected clean traffic, separating uniform and flow-weighted masks.Runtime admission appears below each curve, and the 44 zero-exposure observations are shown separately.Source: Figure 1 · p. 7

Related papers

Loading 2609.25806v1…