Source-linked AI summary
Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting
Bo Chen
TL;DR
The paper asks whether inexpensive prompt-token counts can support model-family attribution through black-box APIs, where broader holdout evidence is limited. It uses a frozen-threshold development/holdout study with validity-gated measurements, finding perfect development separation but incomplete and imperfect holdout generalization. The supported conclusion is that counts fingerprint a shared tokenization stack, not model-family lineage by themselves.
Problem
Black-box attribution needs direct holdout evidence on whether prompt-token-count fingerprints generalize from known endpoint pairs to named model-family membership.
Method
The study uses 24 labeled endpoint pairs split into development and untouched holdout sets, with temporal repeats, controlled texts, a frozen threshold, and validity-gated result states.
Results
Token-count consistency fingerprints a shared tokenization stack, but the standalone family-membership test fails to generalize reliably: development balanced accuracy is 1.00, while strict holdout coverage and eligible-pair sensitivity are each 0.50.
Takeaways & Limitations
Prompt-token counts support narrow endpoint-stack comparisons but are not a reliable standalone necessary test for named model-family membership.
Takeaways & Limitations
Strict validity leaves only four positive and two negative eligible holdout pairs, producing wide Wilson intervals and weakening population-level interpretation.
Abstract
from arXiv · showhide
Black-box model attribution is increasingly relevant when large language models (LLMs) are served through relay and reseller APIs. A tempting low-cost signal is the prompt-token count returned by an OpenAI-compatible endpoint: two models that share a tokenizer and chat template may produce the same count sequence up to a fixed offset. Yet the validity of this signal for broader \emph{model-family} attribution has received little direct holdout testing. We conduct a frozen-threshold study over 24 labeled endpoint pairs, split evenly into a development set and an untouched holdout set, with three temporal repeats and 30 controlled texts per pair. We introduce a validity-gated result contract that distinguishes an observed dissimilarity from an uninformative measurement caused by missing usage data, rate limits, or endpoint policy. The resulting shift-invariant exact-match score perfectly separates the 12 development pairs, yielding a frozen threshold of 0.725. On holdout, however, only 6 of 12 pairs are eligible under the pre-specified three-repeat rule. Among eligible pairs, balanced accuracy is 0.75, sensitivity is 0.50 (95\% Wilson interval 0.15--0.85), and specificity is 1.00 (0.342--1.00). Two same-family pairs---Qwen 3.8 and DeepSeek V4 variants---fall below the frozen threshold. Across 4,320 formal API calls, every log is replayable, while holdout contains 189 non-200 responses and 157 successful responses without prompt-token usage. The study therefore validates token-count consistency as a fingerprint of a shared \emph{tokenization stack}, but rejects its use as a standalone necessary test for model-family lineage.
1 Introduction
The paper tests whether prompt-token-count fingerprints can identify named model-family membership through black-box APIs. It finds that a carefully validated signal can look perfect in development yet fail under strict untouched holdout evaluation.
- Motivation: Prompt-token counts are cheap and deterministic under a fixed request envelope but also include hidden template and system-prompt overhead.This makes them potentially useful for narrow endpoint comparison while risky to interpret as evidence of broader family lineage.
- Research question: The study asks whether a shift-invariant prompt-token-count fingerprint calibrated on known pairs generalizes to named model-family classification.
- Method: A three-state validity contract prevents missing usage or blocked requests from being silently treated as model dissimilarity.The states are informative, uninformative, and error.
- Method: A frozen development/holdout protocol combines an auditable shift-invariant fingerprint, temporal repeats, coverage gates, and replayable per-call logs.
- Results: 1.00 development balanced accuracy falls to 0.50 strict holdout coverage and 0.50 sensitivity among eligible pairs.The result limits what token-count evidence can support for named family membership.
- Results: Rate limits, data-policy restrictions, and responses omitting usage remove half of holdout pairs from strict evaluation.
2 Related Work and Scope
The paper situates token-count fingerprinting among black-box attribution methods and narrows its claim to operational same-family labels. It does not claim ownership verification, stolen-weight detection, or proof of training ancestry.
- Related work: Prior attribution methods use generated queries, output-space comparisons, embedded fingerprints, robustness techniques, or knowledge-boundary behavior.
- Scope: This paper evaluates whether one API-level observable reliably proxies an operational same-family label rather than proving model lineage.
- Threat model: The auditor lacks access to weights, tokenizer files, hidden prompts, provider logs, and routing policy.Endpoints may also omit valid counts, rate-limit requests, or reject routes because of account-level policy.
- Task definition: The operational binary label assigns 1 to the same named model family or base-model variant and 0 to different named model families.
- Task definition: Same-family examples span parameter scales, versions, service tiers, and reasoning or instruction variants, making the task broader than exact-model matching.
- Threat model: Naturally deployed endpoints are evaluated without assuming active spoofing, so success would not establish adversarial security against malicious gateways.
4 Method
The study uses a shift-invariant exact-match fingerprint with strict validity gates and freezes its classification threshold before holdout evaluation.
- 30 controlled texts and temperature=0 requests provide paired prompt-token counts while excluding character-based estimates.Each endpoint receives the same user message, and only integer usage.prompt tokens values are accepted.
- Shift-invariant exact-match fingerprint: The fingerprint compares endpoint count differences and scores the fraction matching the most frequent offset.A fixed overhead, such as a stable hidden chat-template prefix, yields a score of one.
- Validity gates: A repeat is informative only with at least 20 valid paired counts and paired coverage of at least 0.8.Otherwise its score is None rather than zero.
- Validity gates: A pair is eligible only when all three pre-specified repeats are informative, with the pair score equal to their arithmetic mean.This strict rule was fixed before holdout evaluation.
- Threshold calibration: The threshold maximizes development balanced accuracy over observed scores, adjacent midpoints, and boundary candidates, producing τ = 0.725.The calibration artifact is frozen before holdout evaluation.
5 Experimental Design
The experiment uses 24 labeled endpoint pairs split evenly into development and holdout sets, with temporal repeats, replayable logs, and exploratory work excluded from holdout proof.
- Pair construction: 24 endpoint pairs are split before formal collection into 12 development and 12 holdout pairs.Each split contains six operational positives and six negatives.
- Pair construction: The development set includes GLM, Qwen, Gemma, Ministral, and DeepSeek same-family controls plus cross-family negatives.The holdout introduces unseen version combinations and provider pairs.
- Repetition and audit trail: Each pair runs three times, with every request and response logged to an exclusive JSONL file containing replay-relevant metadata.Formal results contain 4,320 calls, evenly divided between development and holdout.
- Repetition and audit trail: The pilot checks engineering only, while P1–P3 from earlier exploratory work remain development data rather than holdout proof.
6 Results
Development separates all pairs under the frozen threshold, but holdout evaluation is limited by strict eligibility and reveals missed same-family pairs and operational non-observability.
- Development results: All 36 development repeats are informative, and every formal call returns HTTP 200 with replayable response logs.
- Development results: At τ = 0.725, development sensitivity, specificity, and balanced accuracy are all 1.00.The clean gap motivates but does not validate the frozen threshold.
- Holdout results: Only 6 of 12 holdout pairs satisfy the strict three-repeat eligibility rule; among them, balanced accuracy is 0.75 and sensitivity is 0.50.Specificity is 1.00, with Wilson intervals reported because the eligible sample is small.
- Holdout results: The two holdout false negatives show that the token-count criterion is not necessary for named family membership.
- Holdout eligibility: 189 holdout responses are non-200 and 157 successful responses omit usage.prompt tokens, while all 2,160 formal holdout logs remain replayable.Catalog availability does not imply that a token-count fingerprint is observable for a specific account and route.
7 Analysis
The shift-invariant count fingerprint cleanly separates development pairs but fails to generalize fully to named model-family membership on holdout. The analysis shows that missing observations and broader within-family variation constrain what token counts can establish.
- Qwen3.8 and DeepSeek V4 same-family pairs fall below the fingerprint threshold, with means of 0.5444 and 0.4333.Their count offsets vary substantially by text, although the experiment cannot identify the responsible mechanism.
- The development gap spans a lowest positive mean of 0.7889 and a highest negative mean of 0.5333.New within-family transformations absent from development create a conceptual distribution shift between tokenization-stack similarity and family membership.
- Strict validity prevents blocked or missing observations from being scored as model failures, but reduces eligible coverage to 50%.Zero denotes observed disagreement, whereas null denotes no valid comparison.
8 Limitations
The study’s conclusions are bounded by operational labels, limited eligible data, one gateway and account, a small corpus, and non-adversarial, non-decomposed measurements. Its reproducibility artifacts are public, but the release and campaign scope remain restricted.
- Operational same-family labels derive from provider namespaces and public version names, not proof of shared training ancestry.They support auditing product claims but are not cryptographic ground truth about training data or parameter descent.
- Only four positives and two negatives remain in the strict eligible holdout, making the Wilson intervals wide.Specificity of 1.0 therefore is not strong evidence of a low population false-positive rate.
- Rate limits, usage omission, routing, and policy are account- and time-dependent because the study uses one gateway and one account.Direct-provider APIs may have different coverage.
- Thirty heterogeneous texts do not estimate language- or domain-specific performance or represent a defined text population.
- The method is diagnostic rather than tamper-resistant because gateways can rewrite usage metadata or emulate another count sequence.
- Tokenizer, template, hidden-prompt, and accounting effects remain entangled, so the score cannot attribute a mismatch to one layer.
- Any revised feature or threshold motivated by holdout failures requires a new untouched holdout.
- The public release retains replayable scoring and failure-classification fields while removing authorization values, headers, call IDs, raw text, and pilot artifacts.The study stopped before the planned full multidimensional campaign after the primary hypothesis failed.
10 Conclusion
The conclusion narrows token-count fingerprinting to consistent tokenization and request-wrapping stacks rather than named model-family lineage. It also recommends frozen evaluation, untouched holdouts, temporal repeats, coverage reporting, and validity-aware outcomes.
- Token-count consistency is useful when the claim is that two endpoints expose a highly consistent tokenization and request-wrapping stack.
- Token counts are not a reliable standalone test of named model-family membership: strict holdout coverage was 50%, with 50% sensitivity among eligible pairs.
- Black-box attribution studies should freeze thresholds, preserve untouched holdouts, repeat observations over time, report observability coverage, and distinguish uninformative measurements from observed dissimilarity.
A Pair Matrix
The pair matrix lists development and holdout endpoint pairs with operational same-family or different-family labels. Development and holdout each contain six positives and six negatives, with holdout including unseen version combinations and provider pairs.
- The development matrix lists six same-family controls and six cross-family negatives across GLM, Qwen, Gemma, Ministral, and DeepSeek.
B Reproduction Commands
The reproduction workflow validates the experiment catalog, runs tokenizer studies separately on development and holdout data, performs frozen-threshold evaluation, and generates an independent summary.
- Setup: Offline regression tests and catalog validation precede the tokenizer experiments.The workflow runs test_fixes.py and validates experiments-v5.yaml with pair-balance checks.
- Experiment: The tokenizer study runs paid API calls for development and holdout roles with three repeats.Both roles use all pairs and tokenizer dimensions, writing results to results/v5.
- Evaluation: A one-shot evaluation applies frozen calibration with a minimum of three informative repeats.The evaluation writes tokenizer-holdout-evaluation-final.json.
- Analysis: The completed results are processed by an independent tokenizer-study analysis script.The final command is analyze_tokenizer_v5.py.