PaperlayerView exact revision on arXiv

Source-linked AI summary

API-TAIL: Detecting Model Substitution in LLM APIs through Temperature-Aligned Likelihood Tails

Haocheng Mei, Yuchuan Tian, Ye Yuan, Chao Xu, Allen Chen, Kai Han, Yunhe Wang

arXiv:2610.36801v1cs.LGcs.CL

TL;DR

Text-only commercial APIs can secretly substitute cheaper models, while legitimate undocumented deployment choices make auditing against one reference unreliable. API-TAIL aligns batch-level typicality before testing the remaining tail, and its evaluation reports controlled false alarms alongside detection of substitution at 200 requests.

  • Problem

    Text-only APIs make partial substitution difficult to distinguish from legitimate output variation caused by undocumented decoding and serving choices, so audits must account for a composite null.

    Source: §1 · p. 1

  • Method

    API-TAIL centers standardized surprisal scores by aligning one effective temperature to the audited batch, then tests the remaining atypical-response count against a threshold calibrated across legitimate configurations.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    False-alarm rates stayed at or below 0.069 on all 26 plain honest endpoints, while power was at least 0.69 in 13 of 16 substitute settings at 200 requests.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On real providers of two model families, API-TAIL accepts every plain honest endpoint and detects 10% substitution with a few hundred requests, at API costs of cents and a few GPU-minutes of local scoring.

    Source: §8 · p. 10

    The composite null covers configurations in C, and validity is evaluated on real honest provider endpoints in the evaluation.

    Source: §2 · p. 3

Abstract

from arXiv · show

Commercial language model APIs return only text, allowing a provider to cut costs by quietly routing a fraction of user requests to a cheaper substitute model while billing for the advertised one. Detecting such partial substitution is challenging because honest providers also alter the output distribution through undocumented decoding and serving choices, such as temperature scaling, sampling truncation, repetition and presence penalties, and default system prompts, forcing an audit to test against a composite null of legitimate configurations rather than a single reference. Our key insight is that these legitimate variations primarily shift response typicality across the entire batch, whereas partial substitution injects individual responses that remain anomalous under the claimed model. Building on this, we introduce API-TAIL, a typicality audit that first absorbs batch-level shifts by aligning a single effective temperature to center standardized surprisal scores across the audited batch, and then flags the endpoint if the number of remaining tail responses exceeds a threshold calibrated across legitimate configurations. Evaluated on real-world commercial APIs, API-TAIL reliably controls false alarms across diverse honest providers while detecting even low-rate partial substitution with a small query budget, substantially outperforming existing rank- and MMD-based audits that lose detection power under the same composite calibration.

1. Introduction

Commercial text-only APIs can secretly replace some advertised-model responses with cheaper substitutes, but undocumented legitimate serving choices make a single-reference audit unreliable. API-TAIL aligns batch typicality before testing the remaining tail, and its reported evaluations show controlled false alarms and detection at low substitution rates.

Problem

  • Undocumented temperatures, truncation, penalties, and system prompts shift honest output distributions, so auditing must test a composite null rather than one reference.A single-reference audit risks false alarms or reduced power when calibrated to accept legitimate deployments.Source: §1 · p. 1

Motivation

  • At 10% substitution with 200 requests, calibrated rank-based and MMD-based audits had zero power, even on controlled short-response panels.Both comparators were calibrated on the same legitimate configurations as API-TAIL.Source: §1 · p. 2

Method

  • API-TAIL aligns one effective temperature to center standardized surprisal scores, then counts remaining atypical responses against a threshold calibrated across legitimate configurations.The procedure scores returned text under the claimed model and uses a score threshold of 3 for tail responses.Source: §1 · p. 2, Figure 1 · p. 2

Evidence

  • False-alarm rates stayed at or below 0.069 on all 26 plain honest endpoints, while power reached at least 0.69 in 13 of 16 substitute settings at 200 requests.The evaluation covered two request protocols, six substitute models served by seven endpoints, and 143,955 responses in total.Source: §1 · p. 2
  • Ablations report power gains of 0.09 to 0.63 from centering instead of maximizing likelihood and 0.62 from full-vocabulary scoring over a top-20 window on long responses.Including penalty configurations in the calibration pool also makes the model developer’s own endpoint pass.Source: §1 · p. 3

2. Problem Setup

The audit assumes access to the advertised model’s weights, although the endpoint reveals only returned text and need not honor sampling parameters. The setup defines substitution and honesty over multiple legitimate configurations, then evaluates audits by false-alarm rate and detection power.

Access model

  • The auditor observes returned texts only, without endpoint log-probabilities, seeds, or logit bias, and does not assume sampling parameters are honored.With the advertised model’s weights, the auditor can compute teacher-forced next-token distributions locally.Source: §2 · p. 3

Partial substitution

  • Partial substitution means independently routing each request to a different model with probability ρ and otherwise answering with the advertised model.Substitutes may be older releases, smaller or mixture-of-experts siblings, or models from another family.Source: §2 · p. 3

Legitimate variation and the composite null

  • Honesty is defined by whether some configuration in C could generate the audited responses, making the null composite rather than a single reference deployment.C includes legitimate choices such as temperature, sampling cutoffs, penalties, system prompts, and inference backends.Source: §2 · p. 3

Validity and power

  • Validity requires false-alarm rates no higher than 0.10 on every evaluated honest endpoint and stability in at least 16 of 20 redrawn calibrations.Power is the probability of flagging substitution at rate ρ for a fixed request budget N.Source: §2 · p. 3

3. API-TAIL: Temperature-Aligned Typicality Auditing

API-TAIL scores responses for typicality under the claimed model, aligns one effective temperature to the audited batch, and tests the remaining atypical-response count against a composite-calibrated threshold. Full-vocabulary deterministic scoring supports reproducible audits.

3. API-TAIL: Temperature-Aligned Typicality Auditing

  • API-TAIL scores each response, aligns one effective temperature to the batch, then compares the remaining atypical-response count with a threshold calibrated across legitimate configurations.The calibration is intended to cover residual deployment differences after alignment.Source: §3 · p. 3

3.1. Typicality under a tempered model

  • A response’s standardized typicality score measures excess surprisal under the tempered claimed model relative to its variance; large positive scores indicate responses the model finds atypical.For samples from the model at the corresponding temperature, the score is approximately standard normal.Source: §3.1 · p. 4

3.2. Temperature alignment

  • The fitted grid temperature centers the batch’s standardized scores, absorbing shared shifts from legitimate decoding and serving choices.Equal-weight centering targets the scores that the test later thresholds, unlike likelihood fitting, which lets long and high-entropy responses dominate.Source: §3.2 · p. 4

3.3. Counting atypical responses against a composite null

  • Because alignment centers batch scores, substitution is detected through an excess of large individual scores, which API-TAIL counts after alignment.The count threshold is the largest per-configuration 95th percentile from simulated honest audits; its false-alarm rate is at most 5% under each configuration, up to simulation error.Source: §3.3 · pp. 4–5

3.4. Exact and reproducible scoring

  • Full-vocabulary scoring accumulates moment sums without truncating to top-k or nucleus tokens, and deterministic GPU kernels make repeated audits bitwise identical.For 200 responses, scoring takes about two GPU-minutes for 30-token responses and about eight for responses up to 400 tokens on one L20X GPU.Source: §3.4 · p. 5

4. Experimental Setup

The experiments test API-TAIL on real commercial endpoints under controlled and user-default request protocols, with simulated partial substitution and a calibration pool spanning legitimate serving configurations.

Claimed models and honest endpoints; Substitutes and simulated audits

  • The evaluation covers 26 plain honest endpoints serving Qwen3.8-27B or Gemma-4-31B-it, plus six substitute models served by seven endpoints.The collected dataset contains 143,955 responses; the honest endpoints are used to evaluate validity.Source: §4 · p. 5

Request protocols and prompts

  • P30 fixes sampling parameters and limits responses to 30 tokens, while P-user uses provider defaults and allows up to 400 tokens.The four panels cross the two models with these two protocols and use 1,000 fresh prompts disjoint from calibration prompts.Source: §4 · p. 5

Substitutes and simulated audits

  • Substitution audits replace each response with a substitute’s response with probability ρ and otherwise use a locally generated response from the claimed model.The study varies ρ across 0.05, 0.10, and 0.25 and audit budgets N across 100, 200, and 400.Source: §4 · p. 5

Legitimate configurations; Baselines; Statistical evaluation

  • Calibration uses 16–19 legitimate configurations per panel, including temperature, truncation, prompt, penalty, and precision variants.Each configuration contributes 2,000 simulated honest audits; the comparisons use fixed pools and thresholds, with paired-bootstrap intervals and Holm correction.Source: §4 · p. 6

5. Substitution Detection under Deployment Variation

Calibrated API-TAIL maintains low false alarms on real honest providers while detecting partial substitution across the main settings and a broad range of substitutes.

Detection at the main setting; Validity on real providers

  • At N=200, API-TAIL’s power for 10% main-substitute detection is 0.33, 0.69, 0.89, and 1.00 across Qwen P30, Qwen P-user, Gemma P30, and Gemma P-user, while its largest false-alarm rates are 0.035, 0.002, 0.003, and 0.069.All evaluated audits stay below the 0.10 false-alarm limit; API-TAIL’s validity verdict holds in all 20 re-drawn calibrations.Source: §5 · p. 6, Table 1 · p. 7

Breadth across substitutes

  • At ρ=0.10 and N=200, API-TAIL reaches power of at least 0.69 in 13 of 16 substitute–panel settings, with a median power of 0.99.Cross-family models are detected at ρ=0.05, while the closest substitute, Qwen3.6-27B, is harder to detect and one deployment remains undetected on long default-sampling responses even at ρ=0.25.Source: §5 · pp. 7–8, Table 2 · p. 7

Design comparisons

  • Table 3 compares API-TAIL with variants that change one design ingredient at a time, including temperature alignment and penalty configurations.Its caption identifies power differences, composite-null thresholds, and false-alarm rates after removing penalty configurations; it does not provide the cell values here.Source: Table 3 · p. 8

Audit budget

  • Doubling the budget from N=200 to N=400 raises power from 0.33 to 0.62 on Qwen P30 and from 0.69 to 0.93 on Qwen P-user.At ρ=0.25, 200 requests yield power from 0.73 to 1.00 across every panel.Source: §5 · p. 8, Figure 2 · p. 8

6. Why the Design Works

API-TAIL’s design choices preserve detection power under legitimate deployment variation: it aligns temperature, scores the full vocabulary, and includes legitimate penalties in calibration.

Aligning, rather than maximizing likelihood

  • Replacing alignment with maximum-likelihood temperature fitting lowers power by 0.22 on Qwen P30, 0.63 on Qwen P-user, and 0.09 on Gemma P30.On Gemma P-user, both audits reach power 1.00.Source: §6 · p. 9

Aligning, rather than fixing the temperature

  • Without temperature alignment, power falls to zero on three panels; alignment yields power 0.33, 0.69, and 1.00 on those panels.On Gemma P30, the aligned and unaligned audits are statistically indistinguishable.Source: §6 · p. 9

Scoring the full vocabulary

  • On Qwen P-user, restricting scores to the top-20 log-probabilities drops power from 0.69 to 0.08, while nucleus-restricted scoring lowers power by 0.30 to 0.83 across three unsaturated panels.The passage attributes the evidence to substituted tokens in the vocabulary tail, beyond top-20 log-probabilities.Source: §6 · p. 9

Covering legitimate penalties

  • Without the five penalty configurations, the model developer’s endpoint has a 0.269 false-alarm rate on Qwen P30; including them reduces the largest rate across plain endpoints to 0.035.An endpoint documenting its own system prompt is flagged in 79% of Qwen P30 and 99% of Qwen P-user audits.Source: §6 · p. 9

7. Related Work

Related audits test model identity, behavioral change, or inference-stack fingerprints, while API-TAIL audits natural traffic against legitimate deployment variation as a composite null. Its typicality score builds on prior likelihood-based text detection but fits temperature to the audited batch and tests an exceedance count.

Auditing which model an API serves

  • Model-equality testing uses MMD, rank-based uniformity tests [37], and prior substitution work compares detection approaches [5]; other studies document deceptive claims and backend divergence [21] [26].Related work also examines behavioral change [7] [29] [8] [6], fingerprints [15] [11] [27] [31] [10] [4], routing dilution [33], spoofing [34], and provider-assisted verification [32] [25] [20].Source: §7 · pp. 9–10
  • Unlike designed-probe or repeated-prompt audits, API-TAIL uses natural traffic and treats legitimate deployment variation [26] as a composite null [1].Source: §7 · p. 10

Likelihood and typicality

  • Fast-DetectGPT standardizes log-likelihood at fixed temperature T=1 [2], whereas API-TAIL fits temperature to the batch and tests an exceedance count against a composite null.The passage says API-TAIL’s ablations show these choices are essential; likelihood, rank, and entropy scoring are established zero-shot detection approaches [14] [30].Source: §7 · p. 10

8. Conclusion

API-TAIL addresses substitution auditing as a composite-null test because providers serve the same model in multiple legitimate ways. It absorbs batch-level shifts, tests the remaining atypical responses, and calibrates their count across legitimate configurations.

8. Conclusion

  • API-TAIL aligns one effective temperature to absorb batch-level shifts from honest decoding and serving choices, then tests the remaining atypical-response count against a pool of legitimate configurations.Source: §8 · p. 10
  • On real providers of two model families, API-TAIL accepts every plain honest endpoint and detects 10% substitution with a few hundred requests.Source: §8 · p. 10
  • The reported audit costs cents in API calls and a few GPU-minutes of local scoring.Source: §8 · p. 10

A. Composite-Null Pool and Endpoints

The appendix describes the locally generated composite-null pool and the endpoints and per-endpoint false-alarm rates reported for API-TAIL.

A. Composite-Null Pool and Endpoints

  • The composite-null pool is generated locally across disjoint prompts, temperatures, top-p and top-k settings, penalties, and precision variants.The passage specifies 1,000×4 Qwen and 500×2 Gemma prompts, with temperatures 0.7 and 1.0, top-p 0.8–1.0, and top-k none, 20, or 64.Source: A. Composite-Null Pool and Endpoints · p. 13, Table 4 · p. 13
  • Endpoints with FP4 weights, system prompts, or a failed 20-prompt temperature probe are classified as special; others are plain.The probe compares T=0 with T=1 on the first 16 tokens using Fisher p<0.05.Source: A. Composite-Null Pool and Endpoints · p. 13
  • Table 5 gives per-endpoint false-alarm rates for API-TAIL at N=200, pool C, and 4,000 audits; it defines “no-𝑇” as ignoring 𝑇 and lists P30 and P-user values.Source: Table 5 · p. 13, A. Composite-Null Pool and Endpoints · p. 13

B. Baselines, Full Power Grid and Reproducibility

The section defines the shared composite calibration and baseline protocols, reports reproducible scoring costs, and presents Table 6’s power grid across substitutes, substitution rates, and audit budgets.

B. Baselines, Full Power Grid and Reproducibility

  • All baselines share the same audit draws and composite calibration.Source: B. Baselines, Full Power Grid and… · p. 13
  • MET has power 0.00 at N_req=200; RUT [37] tests Cramér–von Mises mid-rank uniformity, while Fast-DetectGPT [2] counts fixed-temperature scores above 3.Source: B. Baselines, Full Power Grid and… · p. 13
  • Teacher-forcing scoring is bitwise reproducible; scoring 143,955 responses cost 29.92 USD.The implementation used batch size 1, deterministic kernels, FP64 moments, and 151 temperatures in one pass.Source: B. Baselines, Full Power Grid and… · p. 13
  • Table 6 reports API-TAIL power across substitutes, substitution rates, and budgets of 100, 200, or 400 requests, using 1,000 audits.The supplied caption gives the grid’s scope but not its cell values.Source: Table 6 · p. 13
Loading 2610.36801v1…