Source-linked AI summary
When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline
Stefan G. Creadore, Peyton Woakz
TL;DR
Agent evaluations can measure something different from what their labels imply, and more observations do not resolve that mismatch. This paper audits selected Praxa artifacts and records; its results show how gate status, lifecycle timing, and request-level accounting constrain broader interpretations, while historical provider runs and the full current pipeline remain unreproduced.
Problem
Agent evaluations may record tool status or elapsed time without measuring whether the user’s task was accomplished, so the measured event must match the scientific question.
Source: §1 · pp. 1–2Method
The paper traces selected Praxa measurements to implementation files, historical artifacts, and operational records, then checks descriptive calculations and scoring behavior.
Source: §1 · p. 2, §6 · p. 14Results
A 139-case routing report had zero gating failures despite 27 case failures, while compaction reduction was 94.39% follow-up-only versus 46.54% across the trigger and follow-up calls.
Source: §11 · p. 18Takeaways & Limitations
Gate status, elapsed lifecycle age, and a single follow-up request each measure a narrower construct than case correctness, active execution, or a billing-complete workflow.
Source: §7 · p. 14The audit uses one private system and selected retrospective artifacts; unavailable independence, sampling coverage, task clusters, and repeated-run distributions preclude population-level inference.
Source: §9 · pp. 16–17
Abstract
from arXiv · showhide
Agent evaluations can be numerically correct while measuring a different construct from the one implied by their labels. We present a retrospective measurement audit of selected Praxa AI implementation files, historical evaluation artifacts, and operational records. A 139-case offline routing report contains 112 passes and 27 failures despite zero gating failures, because known gaps are explicitly exempted from the gate. An identifier-free export represents 8,843 tool-attempt rows: 8,395 recorded durations and 448 missing values. Of the durations, 121 equal the signed 32-bit maximum and carry abandoned-client labels; inspected database code clamps elapsed lifecycle age. The pooled recorded 99th percentile is 2,147,483,647 ms, versus 38,118.31 ms among server-observed completed calls. This is a stratum contrast, not a treatment effect. In a documented single-trajectory compaction pilot, the reported follow-up input reduction is 94.39%, but the reduction across the trigger and follow-up calls together is 46.54%. We reproduce the descriptive calculations, verify 91 timing statistics through a separate weighted rational-arithmetic implementation, and execute 13 scoring-function and 12 analysis-verifier tests. Finite-completion bounds show how missing durations limit all-row timing statements without imputing values. The contribution is a source-linked case study and reusable verification package for separating gate policy, lifecycle timing, and request-level accounting from broader agent-performance claims. Historical provider runs and the full current pipeline were not independently reproduced; general capability superiority and population-level statistical significance are not established.
1 Introduction
The paper asks what available Praxa measurements actually establish, noting that recorded events can diverge from task accomplishment. It audits selected artifacts and finds three cases where correct arithmetic does not justify a broader interpretation.
1 Introduction
- The audit traces three cases: a passing development gate alongside failing cases, a latency field containing abandoned-call ages, and token savings that shrink when the trigger call is included.Each case illustrates how arithmetic can be correct while a broader interpretation remains unsupported.Source: §1 · p. 2
- The study connects gate exemptions, a clipped lifecycle-age field, and selective token accounting to source artifacts, computational checks, and finite-denominator bounds.It presents these as evidence and reproducible application, not as newly invented measurement methods.Source: §1 · p. 2
- The proposed measurement contract and controlled follow-up studies have not had their operational effectiveness evaluated.The paper claims no new learning algorithm, state-of-the-art task performance, or experimentally established advantage over retrieval-augmented generation.Source: §1 · p. 2
2 Scope, research questions, and evidence standard
The audit examines selected repository materials and retained historical evidence, framing finite-artifact questions about routing, timing, and compaction. It distinguishes verified calculations from source-reported findings and prospective questions.
2.1 What was inspected
- The audit inspected selected code and documentation, reused dated read-only extracts, and did not run the complete application, historical routing harness, live provider benchmark, or deployed completion test.Selected repository searches do not establish an exhaustive inventory of organizational experiments.Source: §2.1 · p. 2
2.2 Research questions
- RQ1 tests consistency among finite routing artifacts; RQ2 examines which event the recorded latency measures and what missingness permits; RQ3 recomputes compaction under two accounting boundaries.The numerical inputs for compaction are historical documentary evidence, not independently rerun provider measurements.Source: §2.2 · p. 3
2.3 Evidence classes and claim discipline
- RQ4 identifies controls needed for stronger claims, while the audit confines conclusions to verified descriptive calculations and qualified source-supported interpretations.The authors distinguish verified measurements, historical source-reported results, and proposed experiments; “not established” does not mean a capability is impossible or absent.Source: §2.2 · p. 3, §2.3 · p. 3
3 System context and related work
The paper sets its empirical scope at selected system boundaries and distinguishes measurement validity from benchmark rankings. It treats prior work as context, not as evidence for Praxa outcomes.
3.1 Relevant implementation boundaries
- The analysis focuses on selected boundaries whose evidence can be specified precisely; documented or code-defined interfaces are not experimentally validated capabilities.The system architecture description is documentary context, not certification of deployed topology.Source: Table 1 · p. 4, §3.1 · p. 4
- The historical offline routing report cannot establish provider-visible tool admission at the audited revision because it uses a pre-V2 path without model or network calls.Its follow-up category also does not establish full conversational-history processing.Source: §3.1 · p. 4
3.2 Position relative to prior work
- The paper analyzes measurement validity rather than downstream benchmark rankings, using prior studies to distinguish terminal tool labels from verified task outcomes.Its unit is a retained artifact or recorded field, not a newly executed agent task; it does not transfer prior papers’ quantitative results to Praxa.Source: §3.2 · p. 4
- HarnessDev, Repo-To-Skill, and AutoResearch provide prior art in controlled harness evaluation and evidence review, which the paper does not claim to originate or replicate.Other cited work studies distinct constructs, including scientific judgment, collaborative training, and multimodal interaction, none measured here.Source: §3.2 · p. 5
4 Methods and reproducibility design
The methods preserve the boundaries and provenance of historical aggregates and identifier-free timing records. They specify denominators and missingness treatment, then check calculations and scoring behavior with separate implementations and isolated tests.
4.1 Historical aggregate extraction
- For historical acceptance and provider reports, the audit retained reported aggregates and denominators without reconstructing individual cases or unavailable execution timings.The ledger-outage comparison retains its seven-fixture denominator and reported Wilson intervals, but absent candidate code and raw provenance prevent environment-identical replay.Source: §4.1 · p. 5
4.2 Identifier-free operational extraction
- The timing extract groups attempt rows by reporting authority, terminal status, and exact latency value, retaining frequency counts and explicit missing-value bins.It includes all returned rows without success-only filtering; its date span does not prove complete historical coverage or distinguish production traffic from every test or backfill source.Source: §4.2 · p. 5
- The identifier-free frequency aggregation supports univariate descriptive statistics but removes event order, unit linkage, task clusters, repeated-run pairings, and evidence needed to estimate task completion.Identifier removal alone is not a privacy guarantee.Source: §4.2 · p. 6
4.3 Statistics and denominators
- Null durations are counted but not imputed as zero; capped values and observed zeros remain in the primary analysis, while cap deletion is only a sensitivity calculation.Acceptance metrics retain distinct implemented denominators, including case counts, tool-name assertions, and fast-path cases.Source: §4.3 · p. 6
- A separate implementation uses integer sums and rational arithmetic to check weighted timing statistics against the primary outputs, but both implementations depend on the same retained evidence.Finite missingness bounds are sensitivity calculations, not confidence intervals or imputations; their illustrative thresholds are not prespecified service objectives.Source: §4.3 · pp. 6–7
4.4 Newly executed scoring tests
- An isolated harness executed 13 constructed scoring-function fixtures, while a separate verifier tested duplicate detection, null handling, quantile interpolation, cap retention, and bound endpoints.The setup was not the full repository build, type-check, or complete routing runner.Source: §4.4 · p. 7
5 Results
Across the audited artifacts, gate status, recorded timing, and compaction savings depend on measurement boundaries that differ from broader performance claims. Reanalysis and software checks reproduce descriptive calculations, while missing records and historical pilot limits constrain interpretation.
5.1 Offline routing: a gate is not the case denominator
- 139 cases include 112 passes and 27 failures despite zero gating failures, because known-gap failures are exempted from the gate but remain in the denominator.The curated report therefore distinguishes gate policy from its 80.58% case pass rate.Source: §5.1 · p. 7, Table 2 · p. 8
5.2 Isolated scoring conformance
- All 13 original scoring-function tests and 12 verifier tests pass; the separate verifier agrees on 91 timing-statistic cells and eight compaction comparisons.These are software checks, not independent experimental observations or evidence of whole-pipeline correctness.Source: §5.2 · p. 9, Table 3 · p. 9
5.3 Operational latency: lifecycle age in an execution-shaped field
- The pooled p99 is 2,147,483,647 ms, versus 38,118.31 ms for server-observed completed calls; saturated abandoned-client records reflect lifecycle age, not active execution.The capped records are labeled client_never_reported, and inspected finalizer code clamps elapsed lifecycle age; the stratum contrast is not an engineering speedup.Source: §5.3 · pp. 9–10, Table 4 · p. 10
- Among server-observed completed rows, 437 of 7,505 durations are missing, bounding the recorded-field fraction at or below 30 seconds between 92.83% and 98.65%.No missing values are imputed, and the bounds concern the recorded field rather than verified execution or an SLO.Source: §5.3 · p. 11
5.4 Compaction: follow-up savings and two-call accounting
- Follow-up input falls from 10,695 to 600 tokens, a 94.39% reduction; including the trigger gives a 46.54% reduction across the two reported calls.The two-call accounting excludes unreported compaction-only usage and the rest of the trajectory, so total cost savings are not established.Source: §5.4 · pp. 11–12, Figure 3 · p. 12
- The native trigger exceeds the configured limit by 2,598 tokens, and both-arm fact checks do not isolate native-context retention because some information was re-injected.The source says the trigger fails request-envelope qualification; this does not establish that a provider hard context ceiling was exceeded.Source: §5.4 · p. 12
5.5 Secondary historical evidence and absent comparisons
- In a one-weather-task discovery pilot with two executions per arm, answer-TTFT p50 values are 3,008 and 3,264 ms and p95 values are 3,115 and 4,082 ms; superiority is unestablished.Required-read and required-fact checks pass 2/2 in both arms, but raw timings were not recovered.Source: §5.5 · pp. 12–13
- A seven-tool outage artifact reports mutation execution despite failed durable claims in 7/7 baseline fixtures and 0/7 uncommitted candidate fixtures, while product comparison and production extrapolation remain unsupported.The selected database inventory also supplies no controlled current-pipeline comparison; record labels are not independently adjudicated success counts.Source: §5.5 · p. 13
6 Statistical interpretation and attempted falsification
The audit treats its results as exact descriptions of finite artifacts, not estimates from a sampled population. Its checks address alternative explanations, while unresolved provenance and dependence limit broader interpretation.
6 Statistical interpretation and attempted falsification
- The curated corpus lacks a demonstrated sampling mechanism, operational records may share users or tasks, and the compaction pilot has one trajectory with explicit reinjection.Resampling these objects as independent trials would add an assumption the evidence does not justify.Source: §6 · p. 14
- No p-values, standardized treatment effects, or population confidence intervals are reported; statistical significance is not established, and its absence does not establish equivalence.Future paired-trial power calculations would require prespecified effects, outcome variability, dependence structure, and error-rate targets, none supplied by these data.Source: §6 · p. 14
- Alternative-explanation checks cover gate exemptions, saturated timing values, token-accounting boundaries, reinjected facts, arithmetic verification, and missingness bounds.These checks include current clamp logic and abandoned-call labels for saturated values; verifier rejection tests do not replace source attestation or external peer review.Source: §6 · p. 14
- Incomplete falsification attempts leave production-only traffic, historical deployments, task duplication, provider cache effects, and the historical model identifier unresolved.The audit is evidence against unsupported interpretations, not proof that every unfavorable observation reflects an agent defect.Source: §6 · p. 14
7 Discussion: a proposed measurement contract
The three cases show how operational measures can be promoted beyond their event boundaries. The discussion proposes making measurement units and event semantics explicit, while treating the contract’s effectiveness as untested.
7 Discussion: a proposed measurement contract
- Gate status, elapsed age, and follow-up input counts concern release policy, recorded intervals, and individual requests—not necessarily correctness, active execution, or complete workflow accounting.The cases illustrate these distinctions but do not establish how often they occur in other systems.Source: §7 · p. 14
- The proposed contract specifies measurement unit, eligibility, revisions, event boundaries, clock source, reporting authority, terminal status, censoring, and independent outcome evidence.It also includes per-call usage and task or trajectory grouping; production implementation and operational effectiveness are not established.Source: §7 · pp. 14–15
- Execution, queueing, approval waiting, reconciliation, and abandonment age should be recorded as distinct event types.Wide numeric storage and explicit saturation flags distinguish large observed intervals from clipped values; terminal tool status alone does not verify a task outcome.Source: §7 · p. 15
- Acceptance reports should show case outcomes, mandatory-gate outcomes, and assertion-level rates together, keeping exemptions visible in scientific denominators.Previously developed fixtures and newly added categories should be identified so their mixture is not mistaken for a treatment effect.Source: §7 · p. 15
- Compaction accounting should state its boundary in advance and include relevant provider receipts for billing or a measured start-to-terminal interval for latency.Retention checks should distinguish facts in native context from deliberately reinjected facts, which address a different mechanism question.Source: §7 · p. 15
- The source-linked demonstration is the contribution; the proposed contract remains a hypothesis, and the executable verifier is not a production implementation.The application and database schema were not changed.Source: §7 · p. 15
8 Controlled follow-up studies: not run
The proposed follow-up studies specify controlled tests of timing semantics, tool admission, compaction, durable effects, and task completion, but none is reported as run. Their designs prescribe controls and analysis units while acknowledging that sample-size guarantees are unavailable.
Studies not run
- The 36-case smoke selection and broader operator-coverage catalog are prospective only: all scenarios are marked NOT_RUN and provide no executed outcomes or performance estimates.The catalog contains 264 proposed contracts in 27 families and 40 fault overlays.Source: §8 · p. 15
P0: Validate timing semantics
- P0 would test timing-field semantics against independently retained event intervals across controlled completion, waiting, abandonment, reconciliation, and storage-boundary events.The plan first requires deterministic event conformance, then blinded labels for any operational misclassification rate.Source: §8 · p. 15
P1: Ablate current-path tool admission
- P1 would compare dynamic tool admission with a static-tool baseline, changing only admission and measuring independently verified task completion as the primary outcome.The design pairs tasks, counterbalances execution order, and treats other measures as secondary.Source: §8 · pp. 15–16
P2: Evaluate full-trajectory compaction
- P2 would test full-trajectory compaction with paired histories and require evidence of both quality and cost before claiming quality-preserving efficiency.The plan treats trajectories as clusters and calls for provider receipts and wall-clock boundaries.Source: §8 · p. 16
P3: Validate durable effects under faults
- P3 would test durable effects under a defined fault matrix, independently checking external state and preserving missing, duplicate, forbidden, and authorized outcomes.It specifies a provider simulator or authorized sandbox rather than unapproved production mutations.Source: §8 · p. 16
P4: Obtain independently verified task outcomes
- P4 would compare a selected candidate against a matched baseline on held-out realistic tasks, using calibrated verification and reporting every planned arm and run.The primary hypothesis concerns verified task completion, not records whose status field says completed.Source: §8 · p. 16
Design limits
- No numerical power claim or minimum sample-size guarantee is made; sample size depends on target populations and calibration evidence not present in the package.The broader study specifications require a disjoint calibration set, prespecified effect criterion, and declared analysis unit.Source: §8 · pp. 15–16
9 Limitations and threats to validity
The audit’s conclusions are bounded by retrospective artifact selection, imperfect construct alignment, unavailable sampling and repeated-run information, and limited access to the system and source data. These constraints prevent broader causal, statistical, and population-level claims.
Internal validity
- The retrospective audit links inspected database code to historical records but does not reconstruct every deployment or establish the complete causal history.The provider pilot changes compaction behavior and includes reinjection, and artifact selection followed accessible evidence rather than independent sampling.Source: §9 · p. 16
Construct validity
- Routing admission, completed attempt status, verifier-source identity, and schema-token floors are imperfect proxies for task completion, independent adjudication, or full cost.The latency field also combines event semantics, so claims are narrowed to the measured constructs.Source: §9 · pp. 16–17
Statistical conclusion validity
- Unavailable independence, sampling coverage, task clusters, and repeated-run distributions preclude inferential conclusions, while one compaction trajectory cannot estimate population variance.Finite-table missingness bounds do not repair workload selection or restore missing task clusters.Source: §9 · p. 17
External validity
- Evidence from one private agent system and selected internal artifacts does not establish prevalence across frameworks, providers, models, or task populations.The audit did not independently reproduce a standardized task-performance score or fair external-system comparison.Source: §9 · p. 17
Reproducibility validity
- The timing multiset and derived figures are reproducible offline, but non-atomic extracts and shared inputs leave common-source errors unresolved.Full historical provider receipts and acceptance records are not packaged, and the isolated test is not an environment-identical application rerun.Source: §9 · p. 17
10 Reproducibility, ethics, and availability
The archive supports offline checking of retained evidence and code, while clearly distinguishing those checks from unrun historical or live evaluations. It omits direct identifiers and conversation content, but does not claim anonymity or disclose the complete private repository.
Reproducibility
- The companion archive includes the evidence registry, claims ledger, experiment registry, source-derived aggregates, scoring code, tests, analysis scripts, and a second arithmetic verifier.It also contains compact payload digests, query templates, environment receipts, and missingness bounds.Source: §10 · p. 17
Reproduction boundaries
- The historical provider and complete routing commands were documented but not executed, and rerunning SQL against a later database state would create a new extract.Offline reproduction is directed to begin with the packaged data and checksum validation.Source: §10 · p. 17
Availability and privacy
- The package excludes protected prompts, credentials, raw account identifiers, and conversation content, but aggregation does not prove anonymity.Ancillary files retain exact operational frequencies and selected source excerpts without disclosing the complete private repository.Source: §10 · p. 17
Ethics
- AI assistance supported manuscript preparation and review, but additional computational and editorial passes were not independent human-scientist reviews.The authors retain responsibility, and proposed experiments must preserve approval and tenant boundaries in isolated environments.Source: §10 · p. 18
11 Conclusion
The available Praxa evidence supports a measurement case study rather than a general superiority claim: gate, latency, and compaction figures depend on what each measure includes. The appendices identify source artifacts and delimit what reproduced interval calculations can establish.
11 Conclusion
- 94.39% follow-up input reduction becomes 46.54% across the trigger and follow-up calls, while 27 of 139 routing cases fail despite zero gating failures.The pooled latency tail includes capped ages of abandoned client calls, so it is unsuitable as a measure of active execution latency.Source: §11 · p. 18
- The separate weighted-arithmetic implementation confirms 91 timing-statistic cells and eight compaction comparisons, alongside 13 scoring-function tests and 12 verifier tests.These checks clarify what the results measure; stronger capability claims remain contingent on controlled, independently verified task experiments.Source: §11 · p. 18
Appendix A. Evidence identity and source coverage
- Appendix A identifies the evidence registry as the authoritative source mapping and lists selected routing, evaluation, and scoring artifacts with repository identities.The historical routing report and scoring sources are separately identified among the listed artifacts.Source: Appendix A · p. 18
Appendix B. Transparent calculations and interval status
- For seven deterministic fixtures, the reproduced Wilson endpoints are approximately [0.64567, 1] when k = n = 7 and [0, 0.35433] when k = 0, n = 7.The calculation is reproduced, but its conventional binomial interpretation was not established and the intervals are not used as production-safety confidence statements.Source: Appendix B · p. 19
Appendix C. Experimental settings and missing fields
Appendix C distinguishes what the data files represent and leaves unresolved settings unfilled. Appendix D documents finite-completion bounds and independent checks of descriptive calculations.
Appendix C. Experimental settings and missing fields
- The CSVs differ in unit and content: latency_frequencies.csv is a compressed multiset, acceptance_categories.csv contains aggregate categories, and pilot CSVs transcribe historical reports.The validation receipts separate newly executed tests and descriptive checks from unrun follow-up work.Source: Appendix C · pp. 19–20
- Table 7 records reproducibility settings and unresolved fields; omitted fields are not filled with defaults, and artifact-reported model names are not independently verified provider metadata.Source: Table 7 · p. 20
Appendix D. Finite missingness sensitivity and verification
- Finite-completion bounds retain all rows in the denominator and vary missing threshold classifications; the primary analysis does not set missing values to zero.The pooled selection has 8,395 observed and 448 missing durations; the server/completed selection has 7,068 observed and 437 missing.Source: Appendix D · p. 20, Table 8 · p. 20
- The verifier computes weighted quantiles from ordered distinct values and frequencies, using integer sums and rational arithmetic before conversion.Twelve constructed tests include a same-total, different-distribution corruption that must be rejected; tests do not modify source data.Source: Appendix D · pp. 20–21
- Validation receipts record checks and execution environments, but a new check count or regenerated timestamp is not another empirical agent observation.Source: Appendix D · p. 21