PaperlayerView exact revision on arXiv

Source-linked AI summary

IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier

Blake Stenstrom, Charangan Vasantharajan, Brian Sathianathan

arXiv:2609.10494v1cs.CLcs.AIcs.LG

TL;DR

Existing capability benchmarks score advertised model identifiers even though enterprises deploy routes whose serving constraints can change observable capability. IB2 binds and evaluates the deployed route with reliability-inclusive scoring and score-blind adjudication, finding measurable route availability, uneven benchmark discrimination, and a 77.38-to-82.54 endpoint contrast while retaining explicit scope and design limitations.

  • Problem

    Capability benchmarks report model-identifier scores although served routes add constraints that can change observable capability, leaving the measured object unreportable.

  • Method

    IB2 binds the route to the evaluation contract before scoring, keeps first-pass failures in the score, excludes unsupported capability, and makes recovery decisions structurally score-blind.

  • Results

    82.54 versus 77.38 under the same declared upstream revision, precision, and task keys produced a +5.16-point difference with 95% paired interval [0.11, 10.60].

  • Takeaways & Limitations

    Route choice, capability limits, chronology, and evaluator configuration must be part of the reported object rather than hidden behind a model identifier.

  • Takeaways & Limitations

    Four of seven suites saturate, and the bank has no held-out split to confirm that classification against unseen items.

Abstract

from arXiv · show

Enterprises deploy systems, not checkpoints. Usable capability depends jointly on weights, serving route, precision, output contract, and harness, yet all 18 audited benchmarks score advertised model identifiers. We treat this as measurement error and give a protocol that makes it reportable. It has three parts. A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it; a reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out; and adjudication is structurally score-blind. We call the protocol IB2 and release its algorithms, classification tables, request contract, and manifest schemas. Its reference instantiation, 128 locked tasks and 987 assertions over document, spreadsheet, chart, tool and database work, stays sealed: the procedure is the artifact, not the corpus. Across eleven systems, four results. Capability availability is measurable: two complete single-route runs on identical weights later failed distinct predicates of the finalized binding gate, while a third passed that gate before a fresh run. The advertised identifier exposed neither limit. Discrimination is not uniform: four of seven suites saturate under a six-system band, with the spread almost entirely from governed database work and multi-tab joins, so we report interval-backed resolution groups, not ranks; two of the nominal five-label output's four cuts fail multiplicity adjustment. Serving-arm choice moved one declared revision and precision from 77.38 to 82.54, paired interval [0.11,10.60], though the arms differ in access mode, harness generation, and the serving tool-call parser, and harness generation is a property of our evaluator, not any endpoint. Excluding failed responses from denominators changes the point ordering, so reliability inclusion changes a conclusion, not its wording.

1 Introduction

IB2 treats the evaluated object as a deployed route-and-harness system rather than an advertised model identifier. It specifies route binding, reliability-inclusive scoring, score-blind adjudication, and reports route availability, sealed evaluation assets, and explicit scope limits.

  • 1 Introduction: Advertised model identifiers cannot reveal route limits, although serving endpoints impose completion, image, tool-parser, decoding, and precision constraints.The paper scopes route findings to its chosen contract and does not claim documented benchmark misattribution.
  • 1 Introduction: Two routes serving the same weights later failed distinct finalized binding predicates, while a third route passed the complete gate before a fresh run.The first two runs were retrospective relative to the finalized gate; no responses were spliced across routes.
  • 1 Introduction: IB2 binds weights, route, precision, reasoning effort, envelope, tools, output contract, and harness into the scored object.The released contribution includes capability-binding, score-blind recovery, fail-closed comparability, resume, and suite-stratified bootstrap algorithms.
  • 1 Introduction: The reference instantiation contains seven suites, 128 locked tasks, and 987 assertions, while its corpus remains sealed and the procedure is released.IB2 profiles distinguish instantiations, and one withheld route leaves a retained row under-identified and unreproducible in principle by third parties.
  • 1 Introduction: The paper measures a model plus fixed harness on synthetic artifacts through a pinned route, not a production deployment or procurement-grade score.It excludes live retrieval, human-in-the-loop operation, multi-user state, and run-to-run variance estimates.

2 Related work and positioning

IB2 is positioned among benchmarks by separating what is evaluated from how it is measured. Its distinctive capability-benchmark property is route binding, while related work supplies task precedents, reporting practices, and serving-stack precedent.

  • 2 Related work and positioning: OfficeQA Pro is a direct comparator for selectable-text documents and places frontier systems in a lower band on nominally the same capability.The paper presents this difference as informative counter-evidence about bank difficulty, not as evidence that the task family is novel.
  • 2 Related work and positioning: IB2 adopts HELM’s practice of reporting cost and latency beside accuracy and uses a four-field schema for represented activity, setting, work product, and result.These precedents concern reporting discipline rather than route-bound capability measurement.
  • 2 Related work and positioning: MLPerf Inference provides precedent for binding the serving stack, but its closed division reports throughput and latency at a fixed accuracy target rather than measured capability.Its submissions disclose hardware, engine, precision, batching, preprocessing, and quality target.
  • 2 Related work and positioning: The positioning table separates evaluated content from evaluation method and marks route binding, reliability inclusion, and cost or latency reporting.Route bound means verifying the serving route against the evaluation contract before scoring; reliability in score retains first-pass failures.
  • 2 Related work and positioning: IB2 claims novelty for binding the serving route before capability scoring, not for spreadsheet, document, chart, tool-use, or text-to-SQL tasks.Prior benchmarks establish each of those task families, including SpreadsheetBench, DocVQA, ChartQA, ToolLLM, and Spider 2.0.

3 Problem formulation

The paper defines evaluation around a fully specified system under test rather than a model identifier, then distinguishes its score from interval-based claims about comparable task banks. This framing makes route, contract, estimand, and uncertainty explicit.

  • 3.1 The system under test: The system under test binds model identity, serving route, precision, reasoning effort, envelope, tools, output contract, and harness into one tuple.A composed system may route suites to different tuples, but each component must bind independently.
  • 3.1 The system under test: Different serving routes are distinct systems, so scores cannot transfer between routes or be combined into one model row.The paper therefore rejects a model-only leaderboard where serving configuration is not jointly controlled.
  • 3.2 The evaluation contract: The evaluation contract fixes the required envelope and output contract, and Algorithm 1 tests whether a route executes every contract row.The contract must be published because contract executability is itself a reportable result.
  • 3.3 Estimand, population, and what a score licenses: The estimand is one fully specified system’s performance on the locked bank, while the reported score is a census over that bank.The score is distinct from the resampling distribution used for sensitivity analysis across comparably constructed banks.
  • 3.3 Estimand, population, and what a score licenses: The resampling frame treats suite tasks as exchangeable draws from the paper’s constructed generative process, not from all enterprise tasks in the world.The resulting interval is not a claim about run-to-run variance or a universal population of enterprise work.
  • 3.3 Estimand, population, and what a score licenses: Ordinal claims require paired intervals excluding zero; otherwise systems receive a shared resolution group rather than a forced ordering.The protocol does not report p-values or run-to-run variance because its estimand is per-system and each configuration has n=1.
  • 3.4 Work representation: Suite labels alone do not specify represented work, operating conditions, or completion criteria, so Table 3 uses a four-field schema.Its S3 and S4 rows support judging the construct question raised later in the paper.

4 Evaluation protocol

IB2 evaluates enterprise AI systems as served routes, binding capability and governance before locked tasks are released, then scoring reliability-inclusive outputs with score-blind adjudication. The protocol also fixes task-set comparability and reports uncertainty rather than treating every route or score as directly interchangeable.

  • Protocol overview: IB2 maps binding, hashed locked release, gold-blind adjudication, and a score-blind barrier into one evaluation protocol.Adjudication receives an allowlisted projection that excludes gold, grader output, and correctness scores.
  • Binding preflight: Ten binding predicates classify route failures and return BOUND, PARTIALCOVERAGE, or WITHHELD before locked tasks are released.Each predicate carries a failure class and lane set; route incompatibility is not converted into a zero score.
  • Binding preflight: Availability is reported separately as capability availability and governance availability because predicates 1–7 test work execution while predicates 8–10 test disclosure.The protocol therefore avoids collapsing distinct availability dimensions into one statistic.
  • Binding preflight: The PARTIALCOVERAGE branch is unreachable in v0.13.2 because no predicate has the required S4-only lane.Probe 4 covers {S4, S5}, and DeepInfra FP8 was therefore WITHHELD after failing its thirteen-image limit.
  • Adjudication: Every non-OK response enters a typed adjudication ledger whose recovery selector cannot access gold, grader output, or score.A contract-valid but incorrect answer is consequently ineligible for every recovery branch.
  • Scoring: The primary measure is the first-pass, reliability-inclusive, equal-suite IB2-7 Full score, with unsupported tasks excluded rather than scored as zeros.Absent responses earn zero, while parseable contract-invalid responses retain credit for supplied work products; suite weighting prevents larger suites from dominating.

5 The reference instantiation and how it was run

The reference instantiation spans seven enterprise-work suites and uses structured classification, fail-closed run comparability, paired task-set bootstrap intervals, and explicit scope boundaries. Its cohort and execution choices bind observed system behavior to pinned routes and contracts rather than advertised identifiers.

  • Reference instantiation: Seven suites cover spreadsheet, document, chart, tool-calling, and governed-database work across 128 locked tasks and 987 deterministic assertions.The full profile adds image-only document analysis to the Core profile and includes multi-tab reasoning and a ten-million-row SQLite task.
  • Adjudication: Table 5 classifies results from a gold-blind typed projection, using status and coverage fields without access to gold answers, grader output, or scores.Results with coverage state not_run bypass adjudication because they were not attempted.
  • Comparability and resume: Append admission requires the candidate run-plan hash to exactly match the persisted checkpoint, otherwise the work starts a new run without partial merging.Retained task and request hashes are also verified, and duplicates or out-of-plan task identifiers are rejected.
  • Uncertainty: The paired suite-stratified bootstrap resamples identical task keys for both systems and returns a 95% percentile interval for their IB2-7 score difference.The pair-specific seed is canonicalized by sorted identifiers, with reversed displays negating and swapping interval bounds.
  • Difficulty and saturation: Four of seven suites are saturated under the difficulty profile, so the paper treats the taxonomy as unvalidated and requires blind difficulty piloting in future freezes.The paper reports 93.8% of tasks as Expert or Frontier while labels remain effectively constant within suites.
  • Cohort and execution: The eleven-system cohort is purposive rather than exhaustive, and hosted-route observability limits attribution to hidden engines, decoding backends, cache policies, or hardware.Latency includes provider queueing and evaluator-owned work, while cost is treated as a measured run diagnostic rather than a reconciled invoice.

6 Results

The results show that reliability-inclusive, route-aware evaluation distinguishes evidentiary groups, suite discrimination, failure classes, and operational trade-offs more clearly than aggregate scores alone. Interval-backed comparisons merge the nominal top tiers, while suite saturation, route-specific failures, and evaluator-contract choices qualify the interpretation of system capability.

  • Composite results: The primary evidentiary reading merges the nominal top three tiers into one group spanning 88.34 to 68.27 because the two top cuts fail multiplicity adjustment.The T1|T2 and T2|T3 intervals are [0.11, 6.80] and [0.05, 7.62], and neither survives multiplicity adjustment.
  • Suite-level discrimination: Four of seven suites are saturated, two discriminate frontier systems, and S5 is compressed below ceiling rather than evidencing difficulty or saturation.S6, S4, S1 and S3 saturate; S7 and S2 discriminate; S5 has no band system above 90 and ρ = 6.0.
  • Suite-level discrimination: The composite range from 41.10 to 88.34 is driven mainly by governed database work, multi-tab joins, and weaker systems failing S6 outright, not by multimodal document design.S4 is the least discriminative suite, while the two discriminating suites carry the composite range.
  • Reliability and failure evidence: The failure taxonomy records 138 first-pass failures and separates planning, integration, model-behavior, contract, and transport phenomena that the composite conceals.Tool-loop exhaustion is concentrated in GPT-5.6 Luna; contract failures in Kimi K3, Qwen3.6-27B and the composed GLM system; budget exhaustion in the same GLM system.
  • Reliability and failure evidence: First-pass failures remain in the primary score, while recovery values are reported separately and transport corrections do not replace primary scores.Eleven transport failures received exact-request recovery and 82 malformed responses received a permitted supplemental retry.
  • Reliability and failure evidence: Transport attribution is asymmetric: R17 records zero transport failures despite two successful Stage 0 reissues, while other systems lack comparable zero-inclusive counters.The failed pre-envelope attempts, task and suite identifiers were not persisted, so exact attribution is unrecoverable; R17’s score is therefore not directly comparable on this dimension.
  • Route-level diagnostics: The strict output contract makes GLM’s 41.10 a contract-sensitive lower bound for integrations that would accept prose-wrapped responses.The evaluator did not run a permissive-extractor counterfactual, and the strict contract was published before the runs.
  • Operational trade-offs: Among systems with complete cost telemetry, Muse Glimmer 30B has the lowest full-run inference cost at $2.43, while GPT-5.6 Terra has the lowest median task latency at 18.3 seconds.The score-cost and score-latency Pareto frontiers contain different systems.

7 Does the protocol change any conclusion?

The protocol changes what can be concluded by separating route executability, reliability-inclusive scoring, endpoint effects, and statistically supported resolution. It rejects unsupported lanes as measured failures and favors resolution groups over unstable rankings.

  • Executability: Provider P scored 68.27 after passing the full gate, whereas retrospective DeepInfra and CoreWeave runs scored 26.12 and 62.25 before later failing distinct predicates.The first two findings are retrospective because the complete predicate set postdates R12.
  • Executability: The +42.14 and +6.01 contrasts with Provider P are descriptive paired route comparisons, not inferential tier claims.Each route ran once, so no run-to-run variance interval exists.
  • Endpoint effects: 77.38 versus 82.54 produced a 5.16-point endpoint difference with paired interval [0.11, 10.60].The arms also differ in access mode, harness generation, and serving tool-call parser, so the result is not isolated as a route effect.
  • Endpoint effects: Substituting the RunPod arm leaves it in nominal T2 but merges nominal T2 and T3 into one six-member group spanning 78.65 to 68.27.The substitution dissolves a boundary that was already non-robust to interval and multiplicity checks.
  • Reliability: Reliability-inclusive scoring changes the ordering: GPT-5.6 Luna becomes the point leader, while Kimi K3 and Qwen3.6-27B move above systems with higher first-pass reliability.Conditional accuracy is therefore not a substitute for a deployment score.
  • Weighting and coverage: Equal-suite weighting changes scale but not point-estimate positions, while zero-filling unsupported lanes would fabricate measured failures rather than represent withheld capability.The protocol preserves the categorical distinction between unsupported and failed evaluation.

8 Discussion

The discussion frames IB2 as a procedural contribution whose results expose both useful distinctions and limits in the instrument. Its strongest claims concern route binding, failure handling, and saturation, while its corpus, envelope, and evaluation design constrain generalization.

  • Interpretation: A controlled attribution study is future work because the endpoint contrast combines access mode, harness, parser, precision, and serving-route differences.The proposed study fixes the Qwen3.8-27B checkpoint across five serving cells and measures score, validity, executability, and GPU-hours.
  • Discrimination: Four of seven suites are saturated, including S6, S4, S1, and S3, limiting frontier resolution.The reported band medians range from 92.1 to 96.9, and anti-ceiling triggers occur on the same four suites.
  • Discrimination: The reserve for unseen-item confirmation was consumed, so the four-of-seven saturation result is reproducible but not confirmable against unseen calibration items.Three suites retain their classification across checked band sizes, but the reserve limitation remains.
  • Construct scope: OfficeQA Pro places frontier systems much lower on real document collections, while IB2’s S3 leaders exceed 90, indicating that the instruments do not measure the same construct.The paper attributes the difference as likely related to synthetic, bounded packets with controlling evidence always present.
  • Contribution: IB2’s unique contribution is procedural rather than corpus-specific: route binding, gold-blind adjudication, fail-closed resume, and saturation criteria can apply to other task banks.The paper distinguishes the general protocol from its sealed reference corpus.
  • Operational meaning: Contract-validity failures break enterprise machine-readable interfaces, whereas tool-loop exhaustion can degrade gracefully when deployers control the call budget.Unparseable responses remain unrecoverable by consumers.
  • Release boundary: The current release is calibration rather than certification, requiring stronger saturated suites and construct-validity review of S5 before frozen-v1.The release also requires independent review, ambiguity testing, signed artifacts, and corpus governance.

9 Conclusion

IB2 argues that the serving route must be part of the measured system because model-identifier scores cannot express capability availability or distinguish model failure from route refusal. Its results support route-aware reporting while keeping endpoint contrasts and saturation limits explicit.

  • Conclusion: Without route binding, a benchmark cannot express capability availability or distinguish a model that failed from a route that refused.The claim is about reporting what the score describes, not about proving published benchmarks misattributed failures.
  • Conclusion: The protocol binds routes before scoring, keeps failures in the score, excludes unsupported capability, and makes recovery structurally score-blind.The coverage disposition was not exercised, and the only route testing a bound disposition later failed strict-JSON conformance in production.
  • Conclusion: 82.54 versus 77.38 yielded a +5.16-point difference with 95% paired interval [0.11, 10.60], reported as an endpoint effect.The arms also differed in access mode, harness generation, and serving tool-call parser.
  • Conclusion: Four of seven suites saturate under a six-system band, so IB2 reports resolution groups rather than ranks; removing reliability failures from denominators changes point ordering.These findings make reliability inclusion a substantive scoring choice rather than a wording choice.

10 Limitations

The paper’s limitations concern provenance, discrimination, statistical stability, and the chosen evaluation envelope. These boundaries require cautious interpretation of both the benchmark’s results and its broader claims.

  • Threats to validity: The instrument carries unmitigated vendor conflict of interest because its authors designed the scenarios, authored the gold data, and controlled reserve consumption.The eleven third-party systems are held out, but no Iterate system appears in the cohort and the bank was built against Iterate systems.
  • Discrimination: Four of seven suites are saturated, and consumed reserve data prevent confirmation against unseen items.S1 changes classification at band sizes five and seven, so the saturation count is not treated as independently decisive.
  • Statistical stability: Single-run evaluation leaves run-to-run variance unidentified, while the leader pair’s interval is 6.4 points wide against a 0.97-point difference.Resolution groups are a partial response to this instability.
  • Evaluation scope: The thirteen-image, twenty-five-tool, above-32,768-token envelope is a design choice rather than a measured enterprise workload population.Route findings therefore apply to this contract, and a different envelope could admit different routes.

11 Ethics, conflicts of interest, and responsible release

The paper discloses conflicts, reserve consumption, gold-integrity controls, and the limits of a private corpus, while releasing the protocol and audit materials. It also records responsible-release restrictions, funding, vendor noninvolvement, and evaluator-assistance practices.

  • Conflicts and funding: All authors work for Iterate.ai, which funded the study and sells enterprise AI systems, while none of the eleven evaluated systems was an Iterate product.The paper states that no internal result appears and that several evaluated systems are sold by competitors.
  • Reserve consumption: The cohort consumed its private reserve under an attested IB2 Benchmark Maintainer control.Retained configurations cover all seven suites and record the acknowledgment consume-private-reserve.
  • Gold integrity: Gold answers and scoring rules remain locked through versioned manifests, SHA-256 checks, and recorded review for material changes.Published run plans bind the corpus-manifest digest, so an unrecorded change breaks comparison.
  • Private evaluation: The private corpus remains unfalsifiable to readers, although the published protocol can be contested, reimplemented, or applied to an unseen corpus.The authors explicitly state that governance machinery does not resolve the objection to private evaluation data.
  • Responsible release: The methodology, schemas, tooling, ledgers, and derived tables are slated for a deny-by-default, content-addressed release.The release is produced by make verify and each exported file is recorded in MANIFEST.sha256.
  • Responsible release: The paper withholds one provider identity because its public terms prohibit benchmarking and states that vendor agreements may supersede public terms.The authors publish no raw output or provider-level performance claim for that provider and require written permission before naming it.

B.7 Difficulty, admission, and the anti-ceiling policy

The benchmark’s difficulty labels were not validated, so the paper reports the mismatch and applies admission and anti-ceiling rules focused on authentic work and repeated saturation. Its discrimination definition uses ordered selections from a fixed six-system band rather than interpolation.

  • Difficulty: 93.8% of tasks were labelled Expert or Frontier, and the labels were non-predictive of measured difficulty across the realized suites.The authors therefore treat the taxonomy as unvalidated in v0.13.2 and require blind difficulty piloting for future freezes.
  • Admission: Admission requires task and gold consistency, reference conformance, artifact integrity, schema validation, deterministic regrading, and independent gold review for frozen-v1.The paper rejects ambiguity, illegible inputs, undisclosed conventions, and adversarial grading as valid difficulty sources.
  • Anti-ceiling policy: A repeated later-suite mean above 95 triggers saturation review, while a repeated mean above 97.5 prevents that suite from being frozen.The policy does not cap correct scores; revisions must add authentic enterprise constructs in a new benchmark version.
  • Discrimination definition: The discrimination definition uses ordered selections from a six-system band and avoids quantile interpolation because convention choices could change the number of separating suites.The definition is checked in order so a ceiling suite is not called discriminative because one system collapsed.
  • Robustness: Five of seven suite classifications remain invariant across band sizes five, six, and seven, while two move.S3, S4, and S6 stay saturated; S2 stays discriminative; and S5 stays compressed below ceiling.
  • Reported outcome: The paper reports four saturated suites because untrimmed range and mean-based tests would respectively overstate S6 or label S6 and S1 unsaturated.This policy status is reported against the present cohort, including suites where multiple rules are triggered.

C.1 Run disposition

The publication lineage distinguishes completed, superseded, stopped, and published launches without splicing responses across replacements. Run accounting and route diagnostics are reported as descriptive records, including measured cost, latency, and served-route failures.

  • Publication lineage: Seventeen full- or partial-cohort launches comprise eleven published, four superseded, and two stopped launches.No response was spliced across the four full-run replacements.
  • Coverage and validity: Table 14 distinguishes capability-lane admission from Valid responses, with all systems admitted on all seven lanes and no lane withheld.Valid counts tasks that returned a scoreable response, whereas admission is a binding-time property.
  • Run accounting: $245.99 is the displayed cohort inference cost, derived from an unrounded ledger total of $245.9914.The amount is reported as an accounting total rather than a model-performance result.
  • Route disposition: R11 and R12 are complete 128-task runs of identical weights on two routes, and their observed scores were later superseded.The GLM scores are included as served-route contract-failure diagnostics rather than PARTIALCOVERAGE dispositions.
  • Cost and latency: Figure 3 plots measured inference cost and evaluator-observed task latency against score, with empirical Pareto frontiers distinct from score ordering.The figure warns that connected cost or latency frontiers and nominal tier bands should not be confused with score ranking.

D Synthetic scoring-conformance fixtures, one per suite

Seven synthetic fixtures make the scoring protocol inspectable without exposing the private bank, exercising partial credit, tolerances, signs, tool traces, and citation provenance. Their illustrative scores show that the fixtures test failure modes rather than benchmark difficulty or suite-family weighting.

  • Purpose and scope: Seven synthetic fixtures, authored after the reported runs, test transparent protocol conformance without private task text or evaluated-system outputs.They do not validate the private bank’s difficulty, representativeness, or item quality.
  • Route conformance: The fixture materials distinguish serving arms by route components, including a differing serving parser, while keeping the response contract and tool-validation fields byte-identical.The table therefore supports distinguishing systems but does not by itself attribute a score difference to one component.
  • Interpretation boundary: The conformance appendix states that fixture point allocations are illustrative and do not instantiate the sealed bank’s suite-family weighting rule.This prevents fixture totals from being read as estimates of the private benchmark’s representativeness or item quality.
  • Fixture outcomes: Three fixtures pass cleanly and four fail, producing realized totals of S1 100, S2 100, S3 55, S4 100, S5 65, S6 45, and S7 85.The failures exercise partial credit, numeric tolerance, sign handling, tool-trace ordering, and citation provenance.
  • S1: single-tab spreadsheet reasoning: The single-tab spreadsheet fixture scores 100/100 when the response satisfies revenue, region, amount, and sheet-range citation assertions.Its gold values are net revenue 320.00, region West, and regional amount 520.00.
  • S2: multi-tab spreadsheet join: The multi-tab spreadsheet join fixture scores 100/100 for recognized margin, top product, and citations covering both displayed ranges.The fixture computes recognized margin as 140 and identifies P1.
  • S5: chart interpretation: The chart fixture scores 65/100 because reporting +12 misses the gold signed value −12, although the region, direction, and chart citation pass.Assertions are evaluated independently, so the scorer does not reconcile the response’s inconsistent sign and direction.

F Agentic-benchmark best-practice conformance

IB2 reports item-level conformance to the Agentic Benchmark Checklist without converting incomplete evidence into passes. Its material gaps include incomplete independent review, unreleased evaluator access, private corpus and gold, and unestimated label noise or run-to-run variance.

  • Partial status means the mechanism exists but release evidence is incomplete, while N/A means the checklist item does not apply to IB2's grader types.
  • Table 22 records item-level ABC conformance across task validity, outcome validity, and reporting.
  • Independent two-person gold review remains a frozen-v1 requirement despite automated conformance and deterministic regrading.
  • IB2 has not independently audited every irrelevant trace-state field or performed an adversarial reward audit.
  • The sealed corpus and gold remain private, and the controlled private evaluator is not yet deployed for third-party use.
  • Gold-label noise and run-to-run variance are not estimated, although the protocol discloses these as reporting limitations.
Loading 2609.10494v1…