Source-linked AI summary
Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification
Yisen Xi
TL;DR
Anonymous model identity matters for data-handling, supply-chain, and capability decisions, but existing black-box verification lacks validated evidence and self-identification is unreliable. The paper proposes a four-stage forensic audit using archived configuration, catalog fingerprints, cross-length tokenizer differentials, and behavioral corroboration. Retrospective testing produced 7 exact, 2 precision-difference, 1 partial, and 0 counter-directional outcomes, while a flagship prospective case supported the GLM-5.3 version line before official confirmation.
Problem
Anonymous API models require identity evidence for data-handling, supply-chain, and capability decisions, but validated black-box verification methods are absent and self-identification is unreliable.
Method
The paper proposes a four-stage forensic protocol combining archived launch-time configuration, catalog fingerprinting, cross-length tokenizer differentials, and behavioral probes.
Results
7 exact matches, 2 precision-differences, 1 partial match, and 0 counter-directional cases were observed in the retrospective N=10 sample.
Takeaways & Limitations
Measured deployment properties can support a traceable audit trail for due diligence when disclosure is absent or delayed, without claiming certainty.
Takeaways & Limitations
End-to-end blind identification was demonstrated on one flagship case, while confirmation still depends on eventual disclosure and never-revealed models receive graded outputs.
Abstract
from arXiv · showhide
The 2025--2026 AI market has seen a wave of stealth releases: frontier models launched anonymously on developer platforms under codenames. For their users, identity determines data-handling terms, supply-chain risk, and capability expectations. No validated methodology exists for black-box identity verification of anonymous models: practitioner checklists lack accuracy evidence, and self-identification is untrustworthy by design. We propose a four-stage forensic audit protocol for API-served models. Stage 0 reconstructs launch-time configuration from archived platform snapshots (Internet Archive), exposing preview--production drift. Stage 1 fingerprints configuration (context, output ceiling, reasoning, modality) against the platform catalog. Stage 2 tests tokenizer identity with a cross-length differential that rejects short-prompt collisions. Stage 3 corroborates with behavioral probes. We test declaration consistency on 10 known-identity releases (7 exact, 2 precision-differences, 1 partial, 0 counter-directional), not end-to-end identification under anonymity. Identification is validated prospectively on a flagship case whose 2026-08-23 analysis pointed to the GLM-5.3 version line and whose official reveal confirmed those family and version-line inferences (deployment variant was not pre-asserted; Flash was consistent post-reveal), and on three Stage-0-only cases where the protocol produced a graded hypothesis or declined rather than guessed. A standard-library-only implementation is provided as supplementary material.
1. Introduction
Anonymous API models create an identity-verification problem because users need auditable evidence for data-handling, supply-chain, and capability decisions, while self-identification and disclosures may be unavailable or unreliable. The paper contributes a four-stage protocol, retrospective validation, prospective flagship testing, and supporting audit materials.
- The problem: Anonymous models make identity operationally important because data-retention terms, supply-chain risk, and capability expectations depend on the provider and model family.
- The problem: The verification task is to reconstruct model identity from serving-stack artifacts and produce an auditable evidence trail for adoption and compliance decisions.
- Why existing signals are insufficient: Self-identification is unreliable because anonymous models can suppress self-confessions or produce incorrect self-claims, while disclosures may never occur and are not auditable.
- Contributions: The paper contributes a four-stage forensic protocol, including archived configuration forensics, cross-length tokenizer testing, prospective blind validation, and explicit cannot-attribute outcomes.
- Scope and disclosure: The protocol is intended for settings where disclosure is absent, delayed, or disputed, and does not replace disclosure when it exists.
- Validation: The flagship case was re-verified against the known model four days after reveal using a constant tokenizer differential of 75 tokens across four paired probes.
2. Background and Related Work
Prior identification methods rely on cooperative responses, pre-existing fingerprints, or disclosed backends and generally lack anonymous prospective validation. This paper positions its protocol around deployment properties that stealth models cannot easily fake, while systematizing tokenizer differentials and launch-time evidence into a staged audit.
- Existing identification methods: AI-generated-content detection attributes text authorship rather than model provenance, while watermarking and output fingerprinting require cooperation or pre-existing fingerprints.
- Existing identification methods: Active black-box systems classify responses or output statistics, but stealth models are engineered to suppress these signals.
- Serving-integrity audits: Gateway-substitution audits test whether a declared backend is faithful, whereas this paper addresses identity when no truthful declaration exists.
- Gap addressed: Prior work lacks deliberate anonymization validation, declaration-consistency sampling paired with prospective blind testing, and first-class cannot-attribute outputs.
- Tokenizer differentials: Short prompts can create coincidental tokenizer-count matches, and transformed gateway usage requires differential rather than absolute token interpretation.
- Practitioner protocols and community forensics: The protocol systematizes community tokenizer techniques through cross-length constancy, collision rejection, protocolization, and a Stage 0 time-consistent baseline.
- Positioning: The paper differs from related systems through retrospective declaration-consistency testing, prospective blind testing, explicit cannot-attribute outcomes, and a standard-library-only implementation.
3. The Audit Protocol
The protocol builds an auditable identity-verification chain from archived configuration, catalog fingerprints, tokenizer differentials, and behavioral corroboration, while preserving cannot attribute as a valid outcome. Its central safeguard is a mandatory cross-length tokenizer test that rejects short-prompt collisions and supports cross-stage consistency checks.
- Protocol design: Each stage produces artifacts for an auditable evidence chain, and prospective conclusions should be committed before disclosure to strengthen scrutiny resistance.The implementation uses no third-party dependencies, reads keys from the environment, and does not log or store them.
- Stage 0: Configuration-declaration forensics: Stage 0 reconstructs launch-time platform configuration because preview-to-production drift can make current specifications produce false negatives.The resulting baseline determines which declaration is authoritative across snapshots and channels.
- Stage 1: Configuration fingerprinting: Stage 1 carries a conservative context-plus-maximum-output candidate pool into tokenizer testing, while recording modality as an auxiliary signal because declarations can lag serving capability.The flagship target declared video input although its public counterpart did not, so modality divergence was flagged rather than used to discard candidates.
- Stage 2: Cross-length tokenizer differential test: Short-prompt-only constancy yields two true acceptances and three false positives, whereas the mandatory multi-length rule admits only the true family.The full rule is designed to distinguish collisions from identity across paired probes rather than relying on a single short string.
- Stage 2: Cross-length tokenizer differential test: Cross-length differential constancy is necessary for shared-tokenizer attribution but not sufficient, because constant differentials can also arise from collisions or shared same-generation tokenizers.Non-constant differentials implicate different tokenizers, changing injections, or reporting transformations; the converse is not a general theorem.
- Confidence grading and detection boundaries: The protocol declines forced identification when evidence is insufficient and uses Stage 3 as a behavioral cross-check rather than a substitute for deployment-property tests.Configuration and tokenizer layers also support branded-model base attribution and silent-downgrade detection, while inference-stack changes preserving token counts remain invisible.
4. Retrospective Validation
The retrospective validation tests whether launch-time declarations align with later official specifications, while explicitly separating declaration consistency from anonymous end-to-end identification. Across 10 cases, declarations were mostly consistent, and the prospective controls support layered configuration and tokenizer evidence while exposing limits of response-based methods.
- Scope of validation: The retrospective sample validates the evidence channel, not end-to-end identification accuracy under anonymity.Cases required archived launch-time specifications and later known identities, which favors semi-disclosed releases and excludes records lacking either condition.
- Retrospective declaration consistency: 10 cases yielded 7 exact matches, 2 precision-differences, 1 partial match, and 0 counter-directional outcomes.The verdict rubric compares declaration semantics, including official rounding or banding, rather than applying a single raw-integer tolerance.
- Scope of validation: A second same-launch codename was excluded because no independent record-specification snapshot was archived, avoiding double-counting one specification match.The excluded case shared the same true model and launch day as an included case.
- Candidate-pool shrinkage: Stage 1 reduced the flagship catalog from 422 models to 6 using context and max-output equality, then to one with reasoning and modality fields.The protocol carries the six-model L2 pool into Stage 2 for recall preservation while treating modality uniqueness as an auxiliary signal because of declaration lag.
- Candidate-pool shrinkage: The full audit required about 102 API calls and 10^4 prompt tokens, with attribution cost dominated by latency rather than spend.At the flagship’s free-window pricing the cost was $0; at the confirmed model’s list price it was under a few cents.
- Head-to-head comparison: Configuration and tokenizer evidence separated same-family versions, whereas response probes were unstable and failed to separate them in the post-cutoff, out-of-distribution setting.Fourteen of 16 model-probe pairs differed across repeated queries, while Stage 1 reasoning flags and Stage 2 tokenizer tests remained usable without retraining.
5. Prospective Blind Validation
The protocol prospectively attributed Ox Alpha to the GLM-5.3 version line before the official reveal, while preserving uncertainty about the deployment variant. Additional anonymous cases produced graded hypotheses or explicit declines rather than forced identifications.
- 5.1. Flagship case: Ox Alpha (prediction → reveal): Stage 1 reduced 422 catalog models to a unique cross-vendor configuration-equivalence class containing Ox Alpha, GLM-5.3, and glm-latest.Matching context window and maximum output reduced the pool to six; adding modality produced a unique match, while the conservative pool was carried forward for recall.
- 5.1. Flagship case: Ox Alpha (prediction → reveal): Stage 2 found a constant 75-token differential against GLM-5.3 and GLM-5.2 across five texts, rejecting non-family collisions under long-text testing.Version-line separation came from Stage 1, not tokenizer evidence.
- 5.1. Flagship case: Ox Alpha (prediction → reveal): Stage 3 matched GLM serving capabilities through modality and reasoning controls, but its knowledge-boundary signal was ranked below configuration and tokenizer evidence.The knowledge-boundary inference later proved less precise than configuration evidence after the reveal.
- 5.1. Flagship case: Ox Alpha (prediction → reveal): The official reveal confirmed the predicted GLM family and version line; Flash was only a post-reveal consistency, not a pre-asserted deployment variant.The result was supported by dated artifacts and later re-verification against public and vendor-direct endpoints.
- 5.1. Flagship case: Ox Alpha (prediction → reveal): Post-reveal re-verification found an exactly 75-token differential across all four paired probes on both public and vendor-direct endpoints.The fifth canonical short probe was not re-paired, so the study does not claim a five-of-five recheck.
- 5.2. Long-running anonymous case: Big Pickle (two-outcome tracking): The Big Pickle case narrowed the candidate pool to the GLM-5/GLM-5.1 family with low-to-medium confidence, while Stage 2–3 stopped because usable token-usage fields were unavailable.Its two-outcome design preserves a timestamped evidence trail if disclosure never arrives.
- 5.3. “No-answer” demonstrations: attribution boundaries: Three Stage-0-only cases produced graded hypotheses for Cypher and Owl, and a correct cannot-attribute outcome for Aurora.Aurora was deliberately declined because 128K context was common at that time point.
6. Portability and Controlled Adversarial Experiments
Controlled experiments test whether the protocol remains useful when usage fields are forged or backends silently switch. Cross-stage inconsistency alarms expose forged identity signals, while longitudinal tokenizer differentials detect backend changes without vendor cooperation.
- Portability and scope: The configuration and tokenizer layers transfer to adjacent audits where declared-versus-actual divergence is the audit object.The paper presents these experiments as exercising serving-switch and threat-model residuals.
- Controlled adversarial experiments: Forged prompt-token usage produced an apparent Stage 2 acceptance, but Stage 3 identity and marker mismatches triggered a cross-stage alarm.The defense therefore depends on retaining behavioral corroboration rather than relying on Stage 2 alone.
- Controlled adversarial experiments: Backend switching changed a constant Stage 2 differential into non-constant differentials, detecting both simulated swaps.Offline differentials shifted to 71–73, while the live experiment spanned –7 to 23 after switching.
- Portability and scope: Longitudinal Stage 2 is stable when the backend remains fixed and sensitive when token accounting changes, but the experiments simulate switching rather than observe commercial routing policies.The branded-product extension changes the candidate pool and is not demonstrated by a branded case study.
7. Discussion
The protocol is presented as an auditable due-diligence and monitoring method rather than a population-accuracy classifier. Its evidence is strongest within the tested scope, while anonymity, disclosure, channel access, and time-varying platform data constrain conclusions.
- 7.1. Interpretation: The paper claims a validated protocol design supported by one end-to-end anonymous flagship case, criterion-level tests, controlled negatives, declaration validation, and cannot-attribute demonstrations—not a population identification rate.The evidence pyramid is explicitly not a substitute for a multi-model prospective panel.
- 7.2. Limitations: Validation covers free-window stealth models only; the paid-stealth cell is empty.Confirmation still depends on eventual disclosure, and never-revealed models receive graded outputs by design.
- 7.3. Practical use: The protocol delivers snapshots, catalog matches, tokenizer differentials, and graded confidence for adoption decisions and longitudinal serving monitoring.Stages 0–1 can also test whether a served model matches published Model Cards or platform declarations.
- 7.2. Limitations: Future work must expand platform coverage, especially usable usage fields for Zen endpoints, fill the paid-stealth cell, and replicate switch detection inside production gateways.The authors also note that catalogs and terms change, making outputs valid only at their measurement time point.
8. Conclusion
The paper provides a four-stage forensic protocol for anonymous API models, combining retrospective validation, prospective flagship testing, adversarial controls, and an implementation for auditable evidence trails. Its conclusion is deliberately calibrated: measured deployment properties support due diligence, but not certainty when disclosure is absent.
- 8. Conclusion: The protocol combines archived configuration forensics, catalog fingerprinting, cross-length tokenizer differentials, and behavioral corroboration for black-box identity verification.The paper also supplies a standard-library audit tool and evidence archive.
- 8. Conclusion: Declaration consistency was validated on ten known-identity releases, while the flagship case matched the GLM-5.3 family and version line after official reveal.Controlled replays additionally showed alarms for forged usage fields and detection of silent backend switches.
- 8. Conclusion: When disclosure is absent or delayed, deployment measurements support due diligence through a traceable audit trail rather than certainty.The evidence trail includes configuration, tokenizer, and confidence information relevant to supply-chain and data-handling decisions.
9. Availability •
The supplementary materials provide the audit tool, forensic records, controlled-experiment artifacts, evidence archives, and publication-ready figures. The tool implements Stages 1–3, while Stage 0 is documented rather than packaged as a crawler.
- Implementation: The standard-library-only audit tool reads API keys from environment variables without logging or storing them.Its source is included in the supplementary archive.
- Implementation: The CLI implements Stages 1–2 and Stage 3 probes; Stage 0 is documented in Table 2 and Supplementary S3 rather than packaged as a crawler.
- Supplementary materials: Supplementary materials include the stealth-release case corpus, auditor worksheet, and Stage 0 extracts for Ox Alpha and Polaris.
- Experiments: Controlled-experiment logs and scripts cover a usage-forging proxy and a silent serving switch in Supplementary S4.
- Evidence archive: The evidence archive retains 2026-08-23 Ox Alpha measurement artifacts encoding the prospective GLM-5.3-line attribution, plus a deposited Big Pickle case card.
- Figures: Figures 1–4 are supplied as vector PDFs, SVG sources, and 1000 dpi PNG files.
CRediT authorship contribution statement
Yisen Xi contributed across the project’s conceptual, methodological, software, investigative, data, writing, and visualization activities.
- Yisen Xi handled conceptualization, methodology, software, investigation, data curation, writing, and visualization.
Funding
The research received no specific grant funding from public, commercial, or not-for-profit agencies.
- No specific grant supported the research from public, commercial, or not-for-profit funding agencies.