PaperlayerView exact revision on arXiv

Source-linked AI summary

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

Xiangfan Wu, Zonghao Ying, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo

arXiv:2608.16391v1cs.CRcs.AI

TL;DR

Third-party hosted LLM APIs may serve behavior that deviates from a trusted reference, but black-box auditing lacks probability information from the target. Ventor-QTest addresses this with joint AFL and EFL measurements, finding little association with GPQA-Diamond accuracy but pronounced EFL coinciding with declining Terminal-Bench pass rates as task exposure grows.

  • Problem

    Black-box auditing lacks a direct way to measure behavioral deviation between a third-party hosted deployment and a trusted reference.

  • Method

    Ventor-QTest uses repeated constrained-context requests and independent long-sequence probes to jointly estimate AFL and EFL without target-provided probabilities.

  • Results

    Pronounced EFL coincides with declining Terminal-Bench pass rates as task exposure increases, while AFL and EFL show little detectable association with GPQA-Diamond accuracy.

  • Takeaways & Limitations

    AFL and EFL should be reported jointly when auditing long-horizon agentic tasks, while distinguishing their finite probe scope.

  • Takeaways & Limitations

    The audit cannot establish causal effects on downstream task success because probe and benchmark windows were unsynchronized and outcomes have other possible determinants.

Abstract

from arXiv · show

As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \mbox{\textbf{Ventor-QTest}}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports \emph{average fidelity loss} (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report \emph{extreme fidelity loss} (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.

1. Introduction

Hosted LLM APIs create a verification gap because requested model names do not cryptographically attest to the checkpoint or serving configuration. Ventor-QTest addresses this gap with black-box AFL and EFL audits, finding route-specific variation and stronger relevance of pronounced EFL to long-horizon task performance than to GPQA-Diamond accuracy.

  • Verification gap: Hosted API model names are claims rather than cryptographic attestations, while clients typically observe only response strings and provider-controlled metadata.Serving may use an older checkpoint, cheaper substitute, quantized deployment, or different decoding stack.
  • Method: Ventor-QTest formalizes vendor-hosted inference as a route-level stochastic process and distinguishes persistent, intermittent, substitution, and adaptive-routing deviations.The audit is black-box and requires probabilities only from a trusted reference.
  • Method: AFL repeatedly queries frozen constrained contexts, reconstructs categorical target distributions from returned texts, and computes a null-bias-corrected within-window mean coarsened-KL statistic.The repeated-request estimator uses a predeclared finite outcome map.
  • Method: EFL uses independent long-sequence probes to retain the empirical distribution of observed run-level centered-surprisal statistics and expose route-specific variation.Experiments across hosted DeepSeek route snapshots report route-specific EFL.
  • Results: Pronounced EFL coincides with declining Terminal-Bench pass rates as task exposure increases, suggesting greater sensitivity of long-horizon correctness to extreme fidelity loss.This contrast motivates reporting AFL and EFL jointly when auditing long-horizon agentic tasks.
  • Results: AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator across three logprob-capable route conditions.AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy.

2. Scope, Threat Model, and Assumptions

Ventor-QTest audits the served model instance as a stochastic, potentially heterogeneous route against a trusted reference, but its current implementation tests only text generation. The audit detects distributional deviation under explicit assumptions, not model ownership, intent, downstream quality, or perfectly evasive providers.

  • Scope: The audit target is the full served instance, including model version, precision, decoding and serving parameters, and API semantics.Behavioral inconsistency alone cannot establish which component changed or whether the change was intentional.
  • Scope: The implementation tests only the text-generation projection, not tool execution, multimodal preprocessing, or agent-environment interaction.Its finite audit also does not directly test the universal consistency condition and does not imply superior downstream-task performance.
  • Threat model: Ventor-QTest models each route as a stochastic process whose serving windows may vary across replicas, numerical formats, inference kernels, or routing policies.The verifier compares target requests with a trusted reference endpoint under independently sampled audit windows.
  • Threat model: Threat cases include operational deviation, undisclosed quantization, intermittent degradation, substitution or metadata forgery, and adaptive selective routing.Recognizable finite probe sets can be defeated through probe classification, while hidden probes only delay attacks through leakage, repeated observation, or surrogate reconstruction.
  • Assumptions: The audit assumes an authentic, non-colluding reference with matching sampling configuration and valid next-token probabilities, while repeated target requests are approximately independent within route-window cells.The reference must score all 500 prespecified positions for a tail run; nonce freshness prevents exact precomputation but not semantic probe detection.

3. Method

Ventor-QTest is a composite black-box audit that pairs repeated-request AFL with long-sequence EFL, assigning them distinct roles. AFL measures bias-corrected coarsened divergence from text counts, while EFL preserves and summarizes upper-tail run-level deviations.

  • Composite audit: Ventor-QTest combines repeated-request AFL, a within-window coarsened-KL statistic, with long-sequence EFL from the empirical distribution of run-level deviations.The components are reported jointly rather than collapsed into a single scalar.
  • Repeated-request AFL: For each frozen context, a predeclared map assigns every returned string to a finite alphabet, while nonconforming responses map to OTHER.Low-reference-mass categories are deterministically pooled into OTHER before target sampling; the primary threshold is c = 1.
  • Repeated-request AFL: AFL uses a reference-centered Dirichlet correction and subtracts a context-specific finite-sample null baseline estimated from 20,000 parametric-null draws.Negative finite-sample values are retained rather than clipped.
  • Repeated-request AFL: A logprob-derived coarsened-KL comparator provides an auxiliary consistency check, with association tested by context-centred exact permutation and Holm family-wise error control.The comparator never enters the text-count statistic, and significance is limited to the frozen contexts, maps, request configuration, and collection window.
  • Long-sequence EFL: EFL treats complete autoregressive sequences as clustered statistical units, resamples independent runs, and summarizes observed upper-tail behavior rather than estimating KL.Reported summaries include the median, standard deviation, an upper empirical quantile, and the observed maximum.

4. Evaluation

The evaluation compares text-count AFL with a logprob-derived comparator, characterizes EFL across route snapshots, and examines downstream task relationships. Results support jointly reporting AFL and EFL, especially for long-horizon tasks.

  • Evaluation design: The evaluation covers AFL-comparator agreement, EFL revealed by independent runs, and downstream relationships across seven routes.Repeated-request AFL uses three logprob-capable route conditions; sequence probes summarize 20 complete 500-position runs.
  • AFL validation: After removing context fixed effects, Pearson correlation is r=0.971 and Spearman correlation is ρ=0.657 between text-count AFL and the comparator.The six exact route permutations yield p=0.167 for both one-sided tests, so the route-limited result is descriptive rather than confirmatory.
  • EFL characterization: DigitalOcean has the largest median, run-level standard deviation, empirical 0.95 quantile, and maximum EFL summaries, while StreamLake has pronounced EFL despite the smallest median.This shows AFL and EFL need not order route snapshots identically.
  • Downstream relationships: DigitalOcean’s Terminal-Bench pass rate declines from 82.6% in the lowest-exposure quartile to 13.6% in the highest-exposure quartile, while AFL and EFL show little detectable association with GPQA-Diamond accuracy.AFL alone does not order Terminal-Bench success; the concurrence may reflect greater sensitivity of long-horizon correctness to extreme fidelity loss.

5. Related Work

Related work spans model-equality testing, API change tracking, behavioral fingerprinting, capability verification, and query-budget analysis. Ventor-QTest is distinguished by reference-relative distributional effects, separate AFL and EFL measurements, and explicit attention to audit cost and selective-routing limitations.

  • Model equality and API auditing: Prior auditors test model equality, rank uniformity, adversarial substitution, or hardware-backed verification under vendor and production nondeterminism.These approaches include two-sample equality tests, randomized-rank tests, and text- or logprob-based auditing.
  • API change tracking: Ventor-QTest reconstructs a predeclared categorical distribution from repeated text and combines mean coarsened-KL with independent long-sequence empirical-tail measurement.Unlike behavioral fingerprints focused on attribution, identity, or tamper detection, it reports a reference-relative effect without training a classifier.
  • Concurrent work: Two July 2026 concurrent preprints reinforce text-only distributional probes, while Ventor-QTest separately reports within-window AFL and run-level EFL.One Token Is Enough fingerprints models from single-token distributions, whereas IRIS audits whole-stream substitution and routing dilution.
  • Capability verification: Capability benchmarks evaluate deployed task behavior across OCR, multimodal reasoning, software engineering, mathematics, knowledge, tool use, schemas, and agentic coding.KVV applies this capability-verification perspective specifically to hosted deployments.
  • Query budget and monetary cost: 200–250 target calls denote one Gao et al. two-sample test, while LLMmap uses 75 configurations times eight queries for training and testing, separate from 3–8-call verification.The comparison emphasizes that request counts purchase different forms of evidence rather than matching statistical power.
  • Adaptive audit recognition: Public probes create a selective-routing threat, so Ventor-QTest treats its contexts as evidence only for the stated audit distribution rather than as protection through prompt secrecy.Prior work addresses recognizability, mixed or diluted routing, and finite-budget fingerprint imitation.

6. Discussion and Limitations

The discussion limits Ventor-QTest’s measurement scope and cautions against treating AFL or EFL as direct predictors of task success. Observed concurrence between pronounced EFL and declining Terminal-Bench pass rates is suggestive, but benchmark and probe windows were unsynchronized.

  • Measurement scope: AFL measures coarsened KL only on declared maps and within the observed window, whereas EFL measures centered-surprise deviation rather than KL.The sample supports comparison of observed empirical distributions, not precise rare-event probabilities or estimation of X_r(m).
  • Measurement scope: Multinomial calibration assumes within-cell requests are approximately independent and returned reference vectors are fixed.
  • Downstream scope: AFL or EFL has no assumed monotonic relationship with task success, since deviations may hurt, leave unchanged, or improve a particular task.Benchmark outcomes also reflect timeouts, rate limits, tool semantics, and agent-runtime behavior.
  • Downstream scope: Pronounced EFL coincides with decreasing Terminal-Bench pass rate, but unsynchronized probe and benchmark windows limit interpretation of this pattern.

7. Conclusion · A. Audit Procedure · A.1. Outcome Mapping and Reference Support

Ventor-QTest jointly reports AFL and EFL because they capture distinct fidelity-loss observables that can rank routes differently and show different downstream associations. Its frozen, reference-supported procedure maps every response into fixed categories, preserves nonconforming outputs as OTHER, and applies calibrated route-level inference within a finite probe scope.

  • 7. Conclusion: AFL and EFL are noninterchangeable: AFL is a null-bias-corrected within-window coarsened-KL statistic, whereas EFL captures run-level centered-surprisal upper-tail behavior.Experiments show descriptive agreement between AFL and a logprob-derived comparator, while the two measures can order route snapshots differently.
  • 7. Conclusion: Little association appears between either audit output and GPQA-Diamond accuracy, whereas pronounced EFL coincides with declining Terminal-Bench pass rate as task exposure grows.The paper therefore motivates reporting AFL and EFL jointly, while stating the finite probe scope of each inference.
  • A. Audit Procedure: The audit freezes contexts, outcome maps, repetition counts, pooling thresholds, prior strength, multiplicity families, and analysis seeds before target collection.For fixed contexts x_1, . . . , x_J, target routes Q_1, . . . , Q_R, and trusted reference P, these elements define the implemented audit inputs.
  • A. Audit Procedure: Reference probabilities are aggregated through each outcome map, residual mass is assigned to OTHER, and reference-only pooling merges categories with expected count below c.The map is fixed using the reference alone before target querying.
  • A. Audit Procedure: Every target response is mapped to one category; nonconforming text remains in OTHER, no response is deleted, and every request contributes to the multinomial count.The same mapping applies to reference mass, target text counts, and the optional target-logprob comparator.
  • A. Audit Procedure: The procedure computes the audit statistics, saves raw text and metadata, simulates the joint route null, calculates one-sided route p values, and applies Holm correction over the frozen matched-route family.Exposed top log probabilities are stored separately for the logprob-derived coarsened-KL comparator.
  • A.1. Outcome Mapping and Reference Support: Each one-token task declares two to four exact digit labels, with exact labels entering named categories and whitespace-only, multi-token, missing, or otherwise nonconforming outputs entering OTHER.The reference exposes the top 20 token probabilities; declared-label probabilities are retained when present, remaining mass is assigned to OTHER, and the vector is renormalized.

B. Experimental Protocol … B.3. Downstream Evaluation Protocol

The study uses a preregistered, matched black-box audit with repeated one-token requests, long-sequence paired runs, and downstream evaluations on GPQA-Diamond and Terminal-Bench. Its comparator, sequence, and benchmark protocols standardize routes and contexts while documenting exploratory and observational limitations.

  • B. Experimental Protocol: The frozen audit evaluates seven matched routes on 12 shared contexts, with 50 target samples per context–route cell, temperature 1, and a one-token limit.Target log probabilities are not requested from Aliyun, Ark, Baidu, or DeepInfra.
  • B. Experimental Protocol: The matched design contains 4,200 one-token target calls and 12 shared reference calls, while normalized artifacts omit several deployment and request-randomization fields.Omitted fields include provider region, top_p, stream mode, timeout, connection reuse, session identifiers, per-request seed support, and retry information.
  • B.1. Auxiliary Comparator and Sensitivity Inference: Route-averaged primary estimates show Pearson r= 0.989 and perfect rank agreement with the logprob-derived comparator across the matched route conditions.The prior-free statistic also has context-centered Pearson r= 0.926 and Spearman ρ= 0.560.
  • B.1. Auxiliary Comparator and Sensitivity Inference: Sensitivity thresholds c= 1, c= 2, and c= 5 yield centered Pearson correlations of 0.971, 0.942, and 0.951, respectively.Corresponding centered Spearman values are 0.657, 0.653, and 0.755.
  • B.2. Long-Sequence Run-Level Protocol: The long-sequence study uses 20 paired replicates per route, with shared freshly generated nonce challenges within replicates and independent nonces across replicates.Each admitted target sequence contains all 500 prespecified scored positions.
  • B.2. Long-Sequence Run-Level Protocol: The complete sequence is the statistical unit, with pointwise 95% intervals obtained by clustered bootstrap resampling of the 20 runs and no binary stability label.The primary report uses all 20 values at Dᵣ,ᵦ(500) and reports the median, standard deviation, empirical quantiles, and maximum.
  • B.3. Downstream Evaluation Protocol: GPQA-Diamond evaluates all 198 items in its highest-quality subset for each of the seven routes, using shared item order and answer parsing.Reported accuracies use Wilson 95% intervals, while correlation significance enumerates all 7! route-label permutations.
  • B.3. Downstream Evaluation Protocol: Terminal-Bench 2.1 evaluates all 89 tasks per route with the Terminus-2 agent distributed with Harbor 0.21.0 in an exploratory observational study.Collection occurred from 2026-08-13 10:09 to 2026-08-15 02:51 China Standard Time; the study was not a time-aligned continuation of the audit windows.

C. Downstream Benchmark Results

Across downstream benchmarks, route-level fidelity statistics show no detectable association with GPQA-Diamond or Terminal-Bench performance. However, DigitalOcean’s highest-exposure tasks perform substantially worse than its lowest-exposure tasks, with failures concentrated among more request-intensive tasks.

  • GPQA-Diamond: GPQA-Diamond accuracy spans 69.7–74.7%, while median D500 shows no supported route-level association with accuracy.Pearson r=0.347 (exact p=0.446) and Spearman ρ=0.464 (p=0.302).
  • Terminal-Bench: Terminal-Bench pass rate likewise shows no detectable association with Sr.Pearson r=0.081 (exact p=0.873) and Spearman ρ=−0.036 (p=0.963).
  • Exposure analysis: The DigitalOcean Q4 exposure gap is 20.6 percentage points worse than the Q1 gap.The task-bootstrap 95% interval is [−36.0, −4.5] percentage points, with one-sided nonnegative-tail probability 0.0059.
  • Exposure analysis: DigitalOcean’s comparator-relative gaps are +6.1, −30.9, −19.1, and −14.6 percentage points from Q1 to Q4, so the deficit is not monotonic.Exposure is defined using median model-request counts across five matched non-DigitalOcean routes.
  • Exposure analysis: Failed DigitalOcean tasks have a median exposure of 36.5 requests versus 11 requests for successful tasks.The supplied passage reports this contrast while describing task exposure as the median model-request count for the same task across five matched routes.

D. Fixed-sample Request Planning · E. Mean-Component Cost Accounting

Fixed-sample planning defines request budgets around a minimum effect of interest, using conservative multiplicity control and estimator-consistent support pooling. The repeated-request mean component is separately costed using metered contexts, historical token rates, and explicit pricing assumptions.

  • D. Fixed-sample Request Planning: Request budgets are defined relative to a minimum effect of interest through the route’s within-window mean coarsened KL.The planning target is tied to frozen contexts and the route-level mean coarsened-KL quantity.
  • D. Fixed-sample Request Planning: Acquisition planning uses a local asymptotic multinomial likelihood-ratio model with conservative family-wise threshold α*=0.05/R for R simultaneous route comparisons.Approximate power is evaluated under the multiplicity-adjusted threshold.
  • D. Fixed-sample Request Planning: Support changes are incorporated by enumerating integer M after applying the estimator’s same Mπ_ji≥1 reference-only pooling rule.The remaining category count k_j(M) determines the degrees-of-freedom calculation ν(M).
  • D. Fixed-sample Request Planning: At M=50 and ν=24, the 90% planning effect floor is δ=0.0308, while the weakest observed third-party score is 0.1185.The comparison is descriptive because the effect is estimated from audit data; actual route claims use the 20,000-draw protocol-specific null.
  • E. Mean-Component Cost Accounting: The mean-component cost calculation covers only repeated-request acquisition, with 12 exact contexts metered once against official DeepSeek V4 Flash using one output token per request.Table 8 plans J=12, 90% power, family-wise level 0.05, and R=6 route comparisons, with 12 reference distributions shared across routes.
  • E. Mean-Component Cost Accounting: One complete pass uses 432 input and 12 output tokens, with cache-miss rates of $0.14 and $0.28 per million tokens recorded on 2026-08-14.All input is conservatively charged as a cache miss, and the artifact is evidence for these historical rates because the official pricing page is mutable.
  • E. Mean-Component Cost Accounting: The seven-route matched audit costs $0.0224 under the stated assumptions.Cache-hit discounts, provider-specific pricing, retries, taxes, and minimum charges are excluded.

F. Released Experimental Data

The release provides protocols, aggregated response data, route-level analyses, and complete evaluation outcomes supporting reproduction of the paper’s figures, tables, and audits.

  • Released data: The release includes frozen protocols, categorized counts for 4,200 matched-route responses, 12 reference category distributions, context-level estimates, joint-null tests, and pooling sensitivities.These artifacts support analysis of matched-route responses and route-level statistical behavior.
  • Released data: It provides all 20 complete D500 values for each of seven paired routes, convergence summaries, and the task-exposure audit.The sequence-probe data are complete across all seven paired routes.
  • Released data: It includes complete results for 198 GPQA items and all 89 Terminal-Bench task outcomes per route.JSON and CSV artifacts reproduce Figures 2 and 3 and Tables 2, 3, and 6.
Loading 2608.16391v1…