Source-linked AI summary
Real Money, Fake Models: Deceptive Model Claims in Shadow APIs
Yage Zhang, Yukun Jiang, Zeyuan Chen, Michael Backes, Xinyue Shen, Yang Zhang
TL;DR
Frontier-LLM access barriers have produced widely used shadow APIs whose equivalence to official services remains uncertain. The paper audits their prevalence and compares representative services with official APIs across utility, safety, and identity verification, finding substantial performance divergence and verification failures. These findings indicate that shadow APIs cannot be treated as reliable official models, although the audit covers a bounded time window, provider set, and three model families.
Problem
It remains unclear whether widely used shadow APIs faithfully reproduce official API outputs, creating a need to assess their reliability and authenticity.
Method
The study identifies 17 shadow APIs and audits three representative services against official baselines using utility, safety, fingerprinting, and metadata analyses.
Results
Performance diverged by up to 47.21%, safety behavior was unpredictable, and 45.83% of 24 evaluated endpoints failed fingerprint verification.
Takeaways & Limitations
Shadow APIs are unreliable substitutes for official models and can undermine reproducibility when used for research evaluation.
Takeaways & Limitations
The study is a bounded September–December 2025 snapshot, and its provider sample may not represent the full shadow-API market or all LLM families.
Abstract
from arXiv · showhide
Access to frontier large language models (LLMs), such as GPT-5 and Gemini-2.5, is often hindered by high pricing, payment barriers, and regional restrictions. These limitations drive the proliferation of $\textit{shadow APIs}$, third-party services that claim to provide access to official model services without regional limitations via indirect access. Despite their widespread use, it remains unclear whether shadow APIs deliver outputs consistent with those of the official APIs, raising concerns about the reliability of downstream applications and the validity of research findings that depend on them. In this paper, we present the first systematic audit between official LLM APIs and corresponding shadow APIs. We first identify 17 shadow APIs that have been utilized in 187 academic papers, with the most popular one reaching 5,966 citations and 58,639 GitHub stars by December 6, 2025. Through multidimensional auditing of three representative shadow APIs across utility, safety, and model verification, we uncover both indirect and direct evidence of deception practices in shadow APIs. Specifically, we reveal performance divergence reaching up to $47.21\%$, significant unpredictability in safety behaviors, and identity verification failures in $45.83\%$ of fingerprint tests. These deceptive practices critically undermine the reproducibility and validity of scientific research, harm the interests of shadow API users, and damage the reputation of official model providers.
1 Introduction
Shadow APIs have emerged as indirect, lower-cost alternatives to official frontier-LLM APIs, but their reliability and authenticity remain uncertain. This paper systematically audits their prevalence, utility, safety, and model identity.
- Official frontier-LLM APIs impose high prices, payment barriers, and geographical restrictions, motivating third-party shadow services.
- Shadow APIs claim to provide outputs equivalent to official models through indirect access, yet may route and manipulate requests across unauthorized nodes.
- 17 shadow APIs were identified across 187 academic papers, with the most popular reaching 5,966 citations and 58,639 GitHub stars.
- The audit compares shadow and official APIs through utility and safety benchmarking, plus model fingerprinting and output-metadata analysis.
- Gemini-2.5-flash accuracy on MedQA fell from 83.82% officially to approximately 37.00% across examined shadow APIs, while safety scores varied unpredictably.
- Across 24 evaluated endpoints, 45.83% failed fingerprint verification and another 12.50% showed substantial cosine-distance deviations.
2 Preliminary
Shadow APIs are third-party services offering indirect LLM access in restricted regions, emerging from pricing and geographic barriers. Their opacity complicates verification because outputs can be substituted or imitated.
- High frontier-LLM prices and regional restrictions have driven a commercial shadow market offering discounted access without geographical limitations.
- Official terms prohibit API-key resale or redistribution, placing many shadow-api sellers in conflict with service and regulatory requirements.
- Shadow APIs are defined as third-party LLM services providing indirect access in regions where official access is restricted.
- LLM linguistic patterns can serve as fingerprints, and LLMmap actively probes responses to estimate which reference model generated them.
- Model extraction and imitation make it increasingly difficult to distinguish authentic frontier-model outputs from distilled or substituted versions.
3 Landscape of Shadow APIs
Shadow APIs are already widely used in academic research, yet their provider ecosystem is opaque and operationally unstable. The landscape combines substantial scholarly adoption with limited identity, provenance, and infrastructure transparency.
- The study began with 4,183 ICLR 2024 and ACL 2024 papers, yielding 2,113 code-available papers and 92 projects using LLM APIs.
- 17 shadow APIs appeared in 187 research papers, including 116 papers accepted at peer-reviewed conferences or journals.
- The most widely used shadow API accumulated 5,966 citations, while associated repositories received 58,639 GitHub stars.
- Among 17 identified services, 15 lacked transparent identity information or verifiable provenance, and two had already ceased operations.
- Most providers frequently changed upstream model sources without giving users corresponding disclosures.
- Overall, shadow APIs combine widespread popularity with minimal transparency, governance, stable infrastructure, and upstream-model disclosure.
4 Performance Evaluation
The study evaluates shadow APIs against official endpoints across utility and safety benchmarks, finding substantial performance divergence, especially in reasoning and high-risk domains. Safety behavior is also inconsistent across models and attacks, so shadow APIs cannot reliably replace official APIs.
- Science-domain utility: 40.00% and 38.89% are the AIME 2025 accuracy deficits for Gemini-2.5-pro and DeepSeek-Reasoner, respectively, on shadow API A.Shadow APIs A and H also show average accuracy gaps of 9.81% and 6.46%, while GPT-5-mini on GPQA marginally exceeds the official API by 1.18%.
- Sensitive-domain utility: 47.21% is the largest reported performance deficit, occurring for Gemini-2.5-flash on MedQA across shadow APIs.Accuracy falls from 83.82% with the official API to an average of 36.95%, with deficits ranging from 46.51% to 47.21%.
- Sensitive-domain utility: 40.10%–42.73% are the LegalBench gaps between all shadow APIs and official endpoints.Shadow API A additionally shows a 9.98% accuracy drop for DeepSeek-Chat on LegalBench.
- High-stakes reliability: Nearly half of Gemini-2.5-flash shadow-API cases fail to reproduce the official API’s correct answers in high-stakes domains.Reported failures include confusion over HIV diagnostic protocols and misinterpretation of juror-honesty precedents.
- Safety evaluation: 0.90 versus approximately 0.67–0.68 shows the JailbreakBench harmfulness-score gap for Gemini-2.5-flash under FlipAttack.For GPT-5-mini under Base64, shadow API A scores 0.04 versus 0.02 officially; other models and attacks show differing directions of deviation.
5 Model Verification
The paper directly verifies shadow-API model identities using fingerprinting and distributional testing. These tests reveal frequent identity mismatches, statistically distinguishable outputs, and unstable inference behavior.
- Identity mismatches: GPT-4o-mini and GPT-5 responses align with cheaper or different alternatives in several shadow APIs.Examples include GPT-4o-mini resembling Qwen2.5-7B and GPT-5 resembling GLM-4 or DeepSeek-V3; specialized models may also be replaced by non-reasoning models.
- Distributional testing: Reject = True in Model Equality Testing means shadow outputs are statistically distinguishable from official-model outputs.MET tests whether shadow and official outputs come from the same distribution, rejecting equality at α = 0.05.
- Cross-method validation: 74.1% agreement between MET and LLMmap indicates moderate-to-substantial concordance across 54 evaluated cases.The reported Cohen’s κ is 0.512.
- Behavioral signals: Official APIs typically maintain consistent latency and token counts, whereas shadow APIs show irregular spikes and volatility exceeding 2.0× official levels.These operational differences provide an additional signal of inconsistent backend behavior.
- Method validation: LLMmap achieves 96.0% accuracy and MET 88.3% accuracy in a controlled testbed with known honest and deceptive endpoints.Both methods agree on all confirmed substitution cases.
6 Discussion
The discussion links model identity, behavioral consistency, and provider pricing to shadow-API reliability and substitution incentives. It reports that pricing alone does not predict accuracy loss, while several substitution schemes impose economic costs on users.
- Identity and behavior: Matching model identity can coincide with stable behavior, as shown for GPT-5-mini in shadow API E.The paper also reports a structural Gemini-2.5-flash anomaly in which fingerprint fidelity is dissociated from behavioral consistency.
- Predictors of divergence: Pricing has no predictive power for accuracy drop in the OLS analysis across 24 endpoints.The regression uses cosine distance, price ratio, identity mismatch, and a reasoning-model flag as predictors.
- Economic incentives: 7.1–7.25× is the price ratio in an information-premium example where API A advertises Gemini-2.0-flash but delivers Gemini-2.5-flash.The paper distinguishes information-premium, discount-substitution, and resale-markup schemes.
- User losses: 38% of official GPT-5 output volume is delivered by shadow API A despite a 1.00× official price ratio on GPQA.The comparison uses 1,273 queries and imputes equivalent value from actual token counts received.
7 Suggestion
The paper recommends avoiding shadow APIs in research and establishes verification and reporting workflows when direct official access is unavailable.
- Shadow APIs should not be used in research workflows; official APIs are the fundamental solution.
- Auditor Verification Protocol: Auditors should use a four-stage verification pipeline and immediately avoid endpoints that fail any stage.The pipeline includes LLMmap probes and MET with at least 500 samples at α = 0.05.
- Researcher Pre-Registration Checklist: Researchers should preregister endpoint details and confirm that the endpoint passes all four verification stages before data collection.
- Researcher Pre-Registration Checklist: Per-run accuracy, LLMmap fingerprint distance, and MET p-values should accompany results so readers can assess backend reliability.
- Community-Level Actions: Conference organizers should flag undisclosed or unverified third-party endpoints as reproducibility risks, while providers can reduce demand through access and verification improvements.
8 Conclusion
The paper presents a systematic audit showing that shadow APIs exhibit performance divergence and identity inconsistency. It concludes that they cannot be treated as reliable official models.
- Shadow APIs show significant performance divergence and identity inconsistency across providers, benchmarks, and multidimensional evaluations.
- Shadow APIs may fail to reproduce the behaviors of official endpoints, making them unreliable for evaluation.
- Shadow APIs cannot be treated as reliable official models because their claims are deceptive.
Limitations
The study is bounded by its observation window and provider and model coverage. These limits constrain how broadly its measured behavior can be generalized.
- Temporal Scope and Market Volatility: The study observes shadow APIs only from September to December 2025, while providers may change sources, routing, or availability over time.
- Temporal Scope and Market Volatility: Measured behavioral characteristics may drift beyond the observation window because the shadow API market is volatile and opaque.
- Coverage of Providers and Model Families: The 17 selected providers may not represent the full shadow API market, whose other services may use distinct architectures or deceptive strategies.
- Coverage of Providers and Model Families: The audit covers three representative model families rather than all LLM families, limiting direct coverage beyond those families.
Ethical Considerations
The study applies responsible-AI safeguards while evaluating unofficial services, including anonymization, restricted disclosure of harmful outputs, and reproducibility-oriented documentation.
- The study purchased and queried unofficial shadow APIs solely for auditing and transparency, without endorsing their use.
- Audited shadow APIs, academic papers, authors, and institutions were anonymized to expose systemic risks while reducing legal and promotional harms.
- Safety testing used jailbreak attacks, but harmful outputs and exact adversarial prompts were withheld; only aggregated harmfulness metrics were reported.
- The authors followed responsible disclosure practices and reported findings to official providers and affected paper authors.
- Model configurations used seed 42 and temperature 0 when supported, while pricing was normalized to USD per 1 million tokens.
- Compliance and Transparency: Transparent identity requires a legal entity, verifiable registration identifier, and legal documentation.
- Shadow API Selection: Shadow APIs A, E, and H were selected because of popularity, endpoint accessibility, and coverage of GPT, Gemini, and DeepSeek families.
- Discrepancy analysis compares official and shadow responses on MedQA and LegalBench using first-execution results.
C Safety Evaluation on AdvBench
Shadow APIs show safety behavior that diverges unpredictably from official endpoints across AdvBench attacks, while also exhibiting inconsistent latency, token counts, and volatility. These results complicate treating shadow APIs as interchangeable with official APIs.
- Safety behavior: 5× higher harmfulness: Shadow API A scores 0.05 versus 0.01 for the official GPT-5-mini API under the Base64 attack.
- Safety behavior: All Gemini-2.5-flash shadow APIs underestimate risk relative to the official API across attacks, especially FlipAttack.
- Operational behavior: Shadow APIs show inconsistent inference latency and token counts across runs for GPT-4o-mini on AIME and GPQA.
- Operational behavior: Shadow API latency volatility frequently exceeds the official baseline by over 1.2× or 2.0×, especially for Gemini and DeepSeek models.
F Controlled Validation of Detection Methods
The paper validates model-identity and output-distribution detection methods using controlled honest and deceptive endpoints, then relates detection signals to task-performance degradation across shadow APIs. LLMmap performs strongly on controlled identity verification, while MET provides a complementary but less sensitive distribution-shift check.
- Controlled validation: Each controlled validation scenario contains N=100 samples and configures endpoints as either honest or deceptively substituted across GPT, Gemini, and DeepSeek families.
- LLMmap validation: 96.0% overall accuracy: LLMmap records FPR=3.0% and FNR=5.0% when identifying backend models on controlled endpoints.
- MET validation: 88.3% overall accuracy: MET records FPR=4.67% and FNR=18.67% when testing whether outputs match the claimed model’s official distribution.
- Method complementarity: MET detects distributional shifts rather than exact model identity, making it complementary to LLMmap rather than redundant.
- Regression analysis: Price ratio does not predict accuracy drop in either regression specification, so higher-priced shadow APIs do not reliably indicate better output quality.