PaperlayerView exact revision on arXiv

Source-linked AI summary

BTS-AgentBench: A Deterministic, Replayable Pipeline from Read-Only Telemetry Logs to Agent Benchmarks

Jeong-Yoon Kim

arXiv:2608.27334v1cs.CLcs.SE

TL;DR

Industrial telemetry is abundant but rarely compiled into executable multi-turn agent tasks grounded in local facilities. The paper introduces a deterministic pipeline that builds read-only tools, static evidence-grounded tasks, and bounded typed episodes from telemetry. It releases BTS-AgentBench and demonstrates reuse on XAI4HEAT, while defining a deliberately narrow read-only evaluation scope.

  • Problem

    Site-specific telemetry provides local facility vocabulary and observations, but existing benchmarks do not specify how to compile such records into scalable executable multi-turn agent tasks.

  • Method

    The pipeline normalizes telemetry and metadata into read-only tools, compiles static tasks with tool-derived gold answers and evidence, and lifts them into deterministic bounded episodes.

  • Results

    The released BTS-AgentBench contains 532 rows with a 356/87/89 split, while the shared construction path produces 204 XAI4HEAT episodes.

  • Takeaways & Limitations

    Real telemetry can be compiled into agent evaluation while preserving source computations and targets and making interaction contracts, scoring, replay, and provenance inspectable.

  • Takeaways & Limitations

    The benchmark evaluates bounded read-only telemetry tasks, not write-side control, safety-critical actuation, maintenance planning, or long-horizon troubleshooting, and lacks a systematic domain-expert audit.

Abstract

from arXiv · show

Industrial sites contain large volumes of read-only telemetry, but few benchmarks specify how to compile these records into executable multi-turn agent tasks. We present a telemetry-to-episode construction method instantiated as BTS-AgentBench. The pipeline normalizes BTS metadata and raw histories into a read-only tool store, compiles static tasks with tool-derived gold answers and evidence, and lifts retained tasks into typed, bounded operator-facing episodes. The 532-row release adds clarification, goal revision, timestamp policy, quality-gated reporting, and evidence attribution while preserving the source computation and split. Coded contract preflight reports zero findings, and the construction-exclusion controller completes 0/532 rows. Two independent raw-to-episode builds match all 11 logical tool-store exports and reproduce the released 356/87/89 train/dev/test artifact exactly. Applying the shared construction path to XAI4HEAT produces 204 episodes; on its 41-row held-out test split, the controller completes 0 rows and the retained GPT-5.5 execution completes all 41. Code, artifacts, and replay reports are available at https://github.com/kjy7567/BTS-AgentBench.

1 Introduction

The paper presents a deterministic, replayable method for compiling site-specific telemetry into executable, multi-turn agent benchmarks. BTS-AgentBench instantiates this method with grounded tasks, typed interactions, and evidence-based evaluation.

  • Motivation: Industrial telemetry contains site-specific names, relationships, and observations that generic agent data lacks, while manual task construction does not scale.BTS spans three buildings over three years, and its day-window and pairwise candidate spaces reach about 2.19 million and 5.99 million candidates before retention.
  • Method: The proposed method converts fixed telemetry and metadata into read-only tools, executable tasks, operator-facing episodes, source-derived gold answers, evidence, and deterministic scoring targets.The construction is designed so identical inputs and contracts regenerate the same benchmark artifact.
  • Validation: A validation and audit framework provides exact construction replay, deterministic scoring of retained traces, and a rule-based construction-exclusion controller.These mechanisms support replayable benchmark construction and controlled evaluation.
  • Transfer: The same downstream construction and evaluation path is reused on XAI4HEAT after corpus-specific schema mapping, yielding 204 episodes.This tests whether the construction path transfers to a related telemetry corpus rather than relying only on BTS.
  • Positioning: The benchmark separates an agent, user surface, tools, and programmatic evaluation, extending existing interactive tool-use benchmarks from telemetry-derived tasks.Related benchmarks include API-Bank, ToolSandbox, τ-bench, and ACEBench, while this work begins with an existing telemetry corpus.

3 Benchmark Formulation

BTS-AgentBench defines a benchmark unit that preserves telemetry computation while converting retained tasks into bounded, deterministic operator-facing episodes. Its construction pipeline builds read-only task artifacts, composes typed interaction contracts, and validates reproducibility and reporting obligations.

  • 3 Benchmark Formulation: BTS-AgentBench combines a deterministic user simulator, read-only tools, an agent trace, and a programmatic evaluator; success requires telemetry and interaction obligations.No mutable final database state is involved.
  • 3.2 Static Executable Task Layer: Matched BTS metadata and raw histories produce a DuckDB-backed read-only store with stream indexes, quality statistics, aggregates, calendar profiles, and previews.Only metadata points matched to raw history become tool-ready.
  • 3.2 Static Executable Task Layer: Nine task families apply fixed eligibility predicates, including coverage and absolute-mean filters, timestamp-selection rules, and quantile-derived quality rules.Point-disambiguation requires uniquely resolving a target among at least two same-site streams of one point class.
  • 3.2 Static Executable Task Layer: 356/87/89 retained static tasks form the train/dev/test split after diversity-capped selection from combinatorial candidate pools.Candidate counts describe the telemetry search space rather than original BTS task counts.
  • 3.2 Static Executable Task Layer: Each retained row records its query, canonical and alternative calls, structured gold, evidence, verifier, and source metadata under deterministic tie and fallback rules.Nearest lookup selects the earlier observation when absolute offsets are equal.
  • 4.1 Deterministic Episode Compilation: The compiler separates source computation, interaction contract, and surface realization, adding typed clarification, revision, timestamp, quality, commitment, and evidence obligations.The interaction transformations preserve the source computation and split.
  • 4.1 Deterministic Episode Compilation: The finite-state simulator releases missing site or time information only after matching clarification questions, then emits stored revisions, rationale, and evidence requests in order.Agent behavior changes the observed conversation, while user information state and executable targets remain identical across systems.
  • 4.3 Typed Contract Composition and Repair: 41 exact held-out rows are reproduced through fixed reconstruction, while terminal reporting commitment is applied to all 532 retained histories.Independent reconstruction yields the same turns, calls, phase golds, evidence, verifier, and history.

5 Controller-Aware Acceptance

Controller-aware acceptance hardens construction by auditing deterministic shortcuts and excluding rows completed by the phase-complete controller. For BTS-AgentBench, 0/532 rows passed that construction-exclusion audit.

  • Controller audit suite: The controller audit covers state carryover, timestamp policy, aggregation and comparison, quality decisions, and final fields.It uses bounded parsers, explicit site/stream state, read-only tools, and the same evaluator, without a learned parser or LLM.
  • Acceptance rule: A row is eligible only after executable consistency checks, interaction and verifier checks, source-task and split preservation, and controller non-completion.Rows that remain controller-vulnerable are repaired or removed before release.
  • Acceptance outcome: 0/532 rows were completed by the construction-exclusion controller, documenting application of the declared exclusion rather than task difficulty.The controller’s first blocking layers were parse/binding failure for 353 rows, phase-completion failure for 127, and required-tool process failure for 52.

6 Evaluation Setup

The released artifact is validated against a declared executable contract and reproduced through independent replay. Evaluation then runs three frontier agents under shared rows, tools, interaction protocols, and deterministic scoring.

  • Artifact validation: All 532 rows pass coded checks for phase structure, verifier fields, rendered golds, timestamp and quality agreement, targets, evidence identifiers, and prompt–phase alignment.This validates the declared executable contract, not operator realism, corpus representativeness, or task difficulty.
  • Replay validation: Two independent preprocessing executions match all 11 logical tool-store exports, and complete episode builds reproduce the released 356/87/89 rows exactly.Equality covers the generated benchmark artifacts, including turns, calls, phases, and final golds.
  • Evaluation setup: All systems receive the same 89 rows, deterministic user turns, tool descriptions, read-only tools, one-tool-per-turn loop, stopping protocol, and scorer.The evaluator checks clarification, state carryover, temporal alignment, reportability, answer fields, and evidence or rationale follow-up.
  • Frontier model runs: The evaluation compares GPT-5.5, Gemini 3.1 Pro, and Claude Opus 4.7, with one invocation per test row and deterministic rescoring of retained traces.Repeated-call variation is not measured, while retained traces preserve the complete system, user, assistant, and tool exchange.

7 Results

On the released benchmark, success varies across models and task families because complete interaction contracts are stricter than obtaining a telemetry value. Deterministic diagnostic rescoring localizes failures, while the shared pipeline also transfers to XAI4HEAT.

  • Release and test results: GPT-5.5 accomplishes 79/89 test rows, Gemini 3.1 Pro 71/89, and Claude Opus 4.7 58/89.The released benchmark contains 532 rows over nine families with a fixed 356/87/89 split.
  • Family-wise results: 90–100% cells concentrate in direct lookup and aggregation, whereas pairwise comparison is 6/10, 5/10, and 5/10 across the three models.Ranking is 8/10, 5/10, and 4/10; point disambiguation is 8/10, 5/10, and 6/10.
  • Completion contract: Accomplishment requires every applicable clarification, carried-state, revised-goal, timestamp or quality, commitment, evidence, and protocol obligation.An initially correct telemetry value alone is insufficient, and family values are descriptive error localization on this fixed release.
  • Deterministic diagnostic analysis: Table 6 decomposes retained executions into Final, Evidence, Phase, Task, and Protocol scores to expose failures hidden by final-action accomplishment.The decomposition is descriptive and does not estimate repeated-call variance.
  • Cross-corpus portability: The shared path produces 204 XAI4HEAT episodes, with 0 controller completions and 41/41 retained GPT-5.5 accomplishments on its held-out test split.This provides executed portability evidence across the two evaluated telemetry corpora.

8 Discussion

The paper’s primary contribution is a deterministic, replayable method for compiling read-only telemetry into executable, evidence-grounded multi-turn agent benchmarks. BTS-AgentBench preserves source computations while adding typed interaction obligations, enabling inspectable failure analysis, but remains a narrow controlled evaluation.

  • The telemetry-to-episode construction method is the paper’s primary contribution, with BTS-AgentBench as its released benchmark instance.
  • Static tasks fix computation and evidence, interaction contracts specify information state and obligations, and typed phases define deterministic scoring.
  • Exact replay and row-level provenance make construction boundaries inspectable, while retained traces expose state-carryover, policy, phase-completion, and evidence-attribution failures.
  • The method supports portability across BTS and XAI4HEAT after corpus-specific schema mapping, but structurally different logs require additional tools, tasks, and evaluators.
  • Limitations: BTS-AgentBench evaluates bounded read-only telemetry tasks, not write-side control, safety-critical actuation, maintenance planning, or long-horizon troubleshooting.
  • Limitations: The final release lacks a systematic domain-expert audit, and its validation concerns telemetry grounding and executable contracts.

A Release and Rebuild Artifacts

The release and rebuild artifacts implement a deterministic raw-to-episode pipeline that materializes matched telemetry, retains fixed task identities, and derives bounded typed interactions. Replay validates both intermediate stores and released artifacts, while retained cases expose diverse contract outcomes.

  • The replay begins with raw telemetry and metadata, builds read-only tool stores, compiles static tasks, derives typed episodes, and verifies deterministic scoring targets.
  • Only metadata points matched to raw histories become tool-ready, and the retained identity contract fixes 532 evaluated candidates without storing generated conversations or model output.
  • Episode construction withholds selected information, requires clarification and post-answer turns, and carries forward each static task’s computation, gold answer, evidence, and verifier.
  • Validation: All 532 rows pass coded contract preflight checks covering phase structure, verifier fields, runtime policies, targets, evidence identifiers, and prompt–phase alignment.
  • Rebuild validation: Two independently rebuilt stores match all 11 logical exports, and both builds reproduce the 356/87/89 train/dev/test artifact exactly.
  • Retained cases: The retained timestamp-lookup execution receives Final, Evidence, Phase, and Task scores of 1.0 with complete protocol and exact replay matches.

B.2 Case B: Cross-Model Disagreement

Case B shows cross-model disagreement on a point-disambiguation episode whose quality gate supports abstention. The models fail at different trajectory stages: missing stream grounding, later decision underspecification, or complete closure.

  • GPT-5.5 closes the row, while Gemini 3.1 Pro misses early stream grounding and Opus 4.7 under-specifies a later phase-level decision.
  • The quality evidence reports a gap ratio of 1269.5947, supporting abstention rather than reporting the nearest reading as reliable.
  • The main verifier issues are missing stream_id grounding for Gemini and an underspecified phase_4 decision for Opus.
  • The point-disambiguation contract resolves an original point, revises to an alternate, retrieves a nearest reading, checks quality, and requires a final commitment.

B.3 Case C: Shared Hard Failure

Case C is a shared hard failure in which all models largely complete the interaction but fail at final evidence linking or commitment handling. The case distinguishes quality-policy reasoning from contract completion.

  • GPT-5.5, Gemini 3.1 Pro, and Opus 4.7 are each marked partially accomplished for this case.
  • All three models solve most of the interaction but fail at the final evidence-linking stage or final commitment surface.
  • The models disagree about whether quality clears the reporting threshold: Gemini cites observed fraction 0.9958 and gap ratio 4.0119, while Opus reports the reading as-is.
  • The reported interaction issue is invalid_evidence_followup_answer, while the main static issue is a phase_4 decision mismatch.
  • The window-ranking contract ranks an initial and revised month, checks whether the winner changes, quality-gates the revised winner, and requires exact-stream evidence.

B.4 Case D: Near-Success Failure

Case D shows that models can make the correct high-level abstention decision while still failing phase-level obligations required by the benchmark contract.

  • GPT-5.5, Gemini 3.1 Pro, and Opus 4.7 are each labeled partially_accomplished on this row.
  • All three models reach the correct high-level abstention decision, but each leaves a different phase-level gap.The benchmark therefore distinguishes the final action from intermediate clarification, grounding, and comparison requirements.
  • The main static issues are missing left/right mean-value grounding and an incompletely grounded phase_2 comparative cue.
  • The gold contract requires comparing original and revised windows, retrieving the revised winner's nearest reading, applying quality policy, and committing to answer or abstain.

C Evaluation Harness and Runner Configuration

The evaluation harness runs bounded tool-using trajectories against released canonical benchmarks and scores their full traces with deterministic verification across BTS and XAI4HEAT configurations.

  • The Python harness loads a released canonical benchmark, runs one bounded tool-using trajectory per row, records the full trace, and applies the shipped deterministic verifier.
  • The shared BTS test set contains 89 rows, while the XAI4HEAT configuration uses a 41-row test split.
  • All configurations use a deterministic simulator, read-only tools, one-tool-per-turn execution, row-dependent turn budgets, a stopping protocol, and the same scorer.
  • The scorer normalizes natural-language actions, applies explicit numeric tolerances, compares timestamps in UTC, and checks tool arguments and evidence streams separately without an LLM judge.
  • XAI4HEAT adapts substation-channel tables into common site, stream, point, equipment, timestamp, and value fields before materializing its tool store.
  • The XAI4HEAT path reuses the BTS static builders, runtime, audits, controller, runner protocol, and scorer, while retaining corpus-specific parsing and mapping decisions.

D.1 Representative Converted XAI4HEAT Row

The representative XAI4HEAT episode combines clarification, revised temporal goals, nearest-timestamp policy, quality gating, commitment, and evidence follow-up in one bounded interaction.

  • The row asks for an average energy-transfer measurement at XAI4HEAT substation L17 and initially omits the required time detail.
  • The revised goals request the previous day, a nearest reading around 00:04 UTC, a quality-based answer-or-abstain decision, and a final reporting action.
  • The conversion preserves clarification, revision, timestamp policy, quality gating, and evidence follow-up while changing identifiers, values, and domain wording.
  • For November 1, 2019, the previous-day average was 235.7647, but no exact reading existed at 00:04 UTC.
  • The nearest available reading occurred at 07:00 UTC with value 496.0, almost seven hours after the requested time.
  • Although day-level quality met its thresholds with coverage 1.0 and gap ratio 1.0, the final action was to request a more precise timestamp before reporting.
Loading 2608.27334v1…