Source-linked AI summary
Reading Between the Lines: Towards Reliable Black-box LLM Fingerprinting via Zeroth-order Gradient Estimation
Shuo Shao, Yiming Li, Hongwei Yao, Yifei Chen, Yuchen Yang, Zhan Qin
TL;DR
Black-box fingerprinting often relies on outputs that may lose information about model parameters, limiting the distinctiveness of its fingerprints. ZeroPrint estimates input gradients from semantic-preserving text substitutions and uses the resulting Jacobian as a fingerprint. On LeaFBench, it outperforms existing state-of-the-art black-box methods across multiple metrics.
Problem
Black-box fingerprinting commonly relies on model outputs, which nonlinear functions can compress, while extracting parameter-informative features remains a challenge.
Source: §1 · p. 2Method
ZeroPrint estimates input gradients through semantic-preserving word substitutions and uses the estimated Jacobian matrix as a black-box model fingerprint.
Source: §1 · p. 2Results
ZeroPrint consistently outperforms existing state-of-the-art black-box methods across metrics on LeaFBench, with AUC 0.720 and pAUC 0.683 against TRAP.
Source: §1 · p. 2, §5.2 · p. 7Takeaways & Limitations
The paper’s analysis supports gradients as a more informative fingerprint feature than outputs for black-box model auditing.
Source: §1 · p. 2The threat model assumes an illicit copy or unauthorized derivative deployed commercially, with an adversary able to modify the suspicious model.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
The substantial investment required to develop Large Language Models (LLMs) makes them valuable intellectual property, raising significant concerns about copyright protection. LLM fingerprinting has emerged as a key technique to address this, which aims to verify a model's origin by extracting an intrinsic, unique signature (a "fingerprint") and comparing it to that of a source model to identify illicit copies. However, existing black-box fingerprinting methods often fail to generate distinctive LLM fingerprints. This ineffectiveness arises because black-box methods typically rely on model outputs, which lose critical information about the model's unique parameters due to the usage of non-linear functions. To address this, we first leverage Fisher Information Theory to formally demonstrate that the gradient of the model's input is a more informative feature for fingerprinting than the output. Based on this insight, we propose ZeroPrint, a novel method that approximates these information-rich gradients in a black-box setting using zeroth-order estimation. ZeroPrint overcomes the challenge of applying this to discrete text by simulating input perturbations via semantic-preserving word substitutions. This operation allows ZeroPrint to estimate the model's Jacobian matrix as a unique fingerprint. Experiments on the standard benchmark show ZeroPrint achieves a state-of-the-art effectiveness and robustness, significantly outperforming existing black-box methods.
1 Introduction
Black-box fingerprinting relies on observable outputs, which may lose information about model parameters, leaving a gap with white-box methods. ZeroPrint uses estimated input gradients as fingerprints, approximating them through semantic-preserving word substitutions.
1 Introduction
- Black-box fingerprinting trails white-box methods, while output-based approaches risk losing parameter information through nonlinear functions.The introduction frames extracting parameter-informative features as the central black-box challenge.Source: §1 · p. 2
- Fisher Information Theory motivates using the input gradient as a more informative fingerprint feature than the model output.The paper proposes ZeroPrint to capture this gradient in a black-box setting.Source: §1 · p. 2
- ZeroPrint simulates text perturbations with semantic-preserving word substitutions, then uses input-output differences to estimate a model’s Jacobian fingerprint.The estimated fingerprints are compared between suspicious and source models to determine provenance.Source: §1 · p. 2
- ZeroPrint consistently outperforms existing state-of-the-art black-box fingerprinting methods across multiple metrics on LeaFBench.The authors report extensive benchmark experiments demonstrating effectiveness and reliability.Source: §1 · p. 2
2 Background
The background describes autoregressive LLMs and explains fingerprinting as a way to verify model provenance. It contrasts white-box access with API-based black-box methods and identifies the reliability gap that motivates the paper.
2.1 Large Language Models
- LLMs generate text autoregressively, with each token conditioned on previously generated tokens.The paper describes LLMs as computational systems typically defined by their large parameter scale.Source: §2.1 · p. 2
- LLMs’ use across web applications makes protecting their copyright an important challenge.The cited applications include search engines, code generation, and web agents.Source: §2.1 · p. 2
2.2 LLM Fingerprinting
- Fingerprinting verifies model provenance by comparing a suspicious model’s intrinsic signature with that of a source model.The intended audit identifies illicit copies or unauthorized derivatives.Source: §2.2 · pp. 2–3
- White-box methods analyze model parameters, intermediate representations, or backpropagation gradients, but are practically limited to models with accessible parameters [37].These methods include static, forward-pass, and backward-pass approaches [47].Source: §2.2 · p. 3
- Black-box methods require API-level queries and outputs, and include untargeted and targeted fingerprinting [37].Untargeted methods compare response characteristics [34], while targeted methods check whether optimized query-response pairs are reproduced [44].Source: §2.2 · p. 3
- Prior results report a reliability gap: white-box approaches perform effectively through direct access to stable parameters, while output-dependent black-box methods remain less reliable [37].The background presents reliable lineage inference with black-box access as an ongoing challenge.Source: §2.2 · p. 3
3 Motivation and Theoretical Analysis
The section argues that gradients can retain more information about model parameters than outputs, motivating their use as black-box fingerprints. It illustrates the information loss associated with nonlinear outputs and compares the two features using Fisher Information.
3 Motivation and Theoretical Analysis
3.1 The Nonlinearity Bottleneck: A Motivation Example
- In the motivating neural-network example, the gradient retains a parameter-dependent signal, whereas an irreversible nonlinear activation can compress information about W in the output.The example notes GELU and SiLU as nonlinear activations that may lose information about W [15].Source: §3.1 · p. 3
3.2 The Theoretical Edge: Why Gradients Carry More Information
- The analysis uses Fisher Information to compare how much the observable gradient D and output Y reveal about parameter W.It treats greater Fisher Information as indicating more information about the parameter.Source: §3.2 · pp. 3–4
- Under the theorem’s stated assumptions and first-order approximation, I_D(W) > I_Y(W), so the gradient contains more information about W than the output.The theorem considers a nonlinear, non-affine activation, normally distributed input, and f′′(K) ≠ 0.Source: §3.2 · p. 4
4 Methodology
ZeroPrint approximates input gradients in a black-box setting to use them as LLM fingerprints. Its methodology specifies the audit setting, constructs semantically perturbed queries, estimates Jacobians, and compares fingerprints.
4 Methodology
- ZeroPrint uses zeroth-order estimation to approximate gradients, which are more informative for fingerprinting than outputs, without direct access needed for backpropagation.The method adapts gradient estimation to black-box LLM fingerprinting.Source: §4 · p. 4
4.1 Threat Model
- The adversary may fine-tune or prompt-modify a deployed illicit derivative, while the auditor has API-only access and a query limit for each model.The auditor can submit inputs and observe textual outputs but cannot access internal states or logits.Source: §4.1 · p. 4
4.2 Overview of ZeroPrint
- The workflow constructs a query set, estimates and aggregates a Jacobian fingerprint from model inputs and outputs, then compares source and suspicious fingerprints for attribution.Fingerprint comparison uses a similarity score to determine provenance.Source: §4.2 · p. 4, §4.4 · p. 6
4.3 Query Set Construction
- Base queries use code-completion prompts for compatibility across models, and semantic-similarity word substitutions create perturbations for gradient estimation.The query set contains n base queries and m perturbed queries per base query.Source: §4.3 · p. 5
4.4 Fingerprint Generation via Zeroth-order Gradient Estimation
- The method embeds inputs and repeated outputs, regresses output differences on input differences with ridge regularization, and averages per-query Jacobians into the model fingerprint.Repeated responses are used to mitigate output randomness; ridge regression addresses instability when the estimation system is underdetermined.Source: §4.4 · pp. 5–6
5 Evaluation
ZeroPrint is evaluated against black-box and white-box baselines, then examined across query settings, query data, embedding models, adaptive attacks, and runtime. It achieves strong benchmark performance under a 200-query limit, with results varying by query configuration and data source.
5.1 Experimental Settings
- The benchmark uses 149 LeaFBench model instances, compares four black-box baselines and white-box REEF, and limits queries to 200 per model.The benchmark includes seven foundation models and post-development variants; TRAP assumes black-box access to the suspicious model and white-box access to the source model.Source: §5.1 · p. 6
5.2 Main Results
- ZeroPrint achieves state-of-the-art or comparable performance across evaluated metrics, significantly exceeds untargeted methods, and outperforms TRAP with AUC 0.720 and pAUC 0.683.A performance gap remains against white-box REEF; these results use a limit of 200 queries per model.Source: §5.2 · p. 7
5.3 Ablation Study
- Performance stays relatively stable across 1–5 base queries, while increasing perturbed queries improves results, peaking at m=14 with AUC 0.778 and pAUC 0.706.With repeats per query, AUC is 0.607 at t=1 and reaches 0.720 at t=20, after which results plateau and fluctuate slightly; Figure 2 varies n, m, and t.Source: §5.3 · p. 7, Figure 2 · p. 7
- HumanEval code-completion queries outperform the tested QA and Wikipedia data, whose less structured prompts produce less consistent derivative behavior.The results are reported in Table 2; the authors attribute the difference to more uniform responses elicited by structured code-completion prompts.Source: §5.3 · p. 7, Table 2 · p. 7
- Embedding dimension trades off semantic representation against Jacobian estimation accuracy: dimensions that are too small or too large degrade effectiveness.The larger matrix is harder to estimate under query constraints; MPNet and EmbeddingGemma have the same dimension and comparable results (Table 3).Source: §5.3 · p. 8, Table 3 · p. 8
5.4 Resistance to Potential Adaptive Attacks
- ZeroPrint maintains high fingerprint similarity under input paraphrasing and output perturbation attacks, with robustness attributed to semantic differences, embedding averaging, and gradient-based fingerprints.Table 4 summarizes fingerprint similarities for attacked versions; averaging filters random zero-mean noise, and gradient fingerprints are less sensitive to small random changes in absolute outputs.Source: §5.4 · p. 8, Table 4 · p. 8
5.5 Overhead Evaluation
- Fingerprint extraction and verification take 424.9 seconds on average, faster than most black-box baselines but slower than REEF and LLMmap.The evaluation covers a subset of four LLMs; the authors report that fingerprinting one model completes in under two minutes on average.Source: §5.5 · p. 8, Figure 3 · p. 8
6 Conclusion
The conclusion presents ZeroPrint as a black-box LLM fingerprinting method that estimates input gradients, which the paper’s Fisher Information analysis identifies as more informative about model parameters than outputs. Experiments report state-of-the-art effectiveness and robustness against existing black-box methods, with practical overhead.
6 Conclusion
- Fisher Information analysis indicates that a model’s input gradient contains significantly more information about its parameters than its output.The paper therefore treats gradients as a superior feature for fingerprinting.Source: §6 · p. 8
- ZeroPrint estimates information-rich gradients in a black-box setting by simulating text-input perturbations with semantic-preserving word substitutions.The method approximates gradients to produce a model fingerprint.Source: §6 · p. 8
- Experiments report state-of-the-art effectiveness and robustness over existing black-box methods, with practical overhead.Source: §6 · p. 8
A Proof of Theorem 1
The proof analyzes Fisher information for outputs and input gradients under a Gaussian-input model, using a first-order Taylor approximation for the gradient. It invokes the normal-distribution Fisher information formula and the Data Processing Inequality in its derivation.
A Proof of Theorem 1
- The proof introduces the normal-distribution Fisher information formula for a variable whose mean and variance depend differentiably on a parameter.Source: A Proof of Theorem 1 · p. 10
- Theorem 1 considers a nonlinear, non-affine function of a linear transform of a zero-mean normal input, with a nonzero second derivative at K.It compares Fisher information about W in the output Y and input gradient D under a first-order Taylor approximation.Source: A Proof of Theorem 1 · p. 10
- Because Y is a parameter-independent transformation of Z, the proof applies the Data Processing Inequality to bound information in Y.Source: A Proof of Theorem 1 · p. 11
- The gradient analysis neglects the Taylor remainder to obtain an approximate linear form before deriving its mean and variance as functions of W.The proof then differentiates those quantities and invokes the normal-model Fisher information formula.Source: A Proof of Theorem 1 · p. 11
- The appendix also gives an algorithm that forms base queries and creates perturbed queries by replacing selected words with randomly chosen semantically similar candidates.It returns the combined query set.Source: A Proof of Theorem 1 · p. 11
B Implementation Details
The implementation details specify the benchmark, ZeroPrint query construction and fingerprint-generation inputs, and comparison settings for baselines and additional studies.
B Implementation Details
- LeaFBench covers 7 base-model lineages, 13 post-development techniques, and 149 model instances.The listed techniques include tuning, quantization, merging, distillation, prompting, sampling, retrieval augmentation, and adversarial manipulation.Source: B Implementation Details · p. 11
- ZeroPrint forms base queries from the first 20 words of HumanEval code snippets and replaces 3 words per query with one of the top-10 similar words from GloVe.The coefficient for Ridge Regression is specified in the following text only as an incomplete fragment.Source: B Implementation Details · p. 11
- The fingerprint-generation algorithm takes a black-box target model, perturbed queries, a sentence embedding model, repetition count, and Ridge regularization, and outputs an aggregated Jacobian fingerprint.It estimates local Jacobians using input and output embedding differences.Source: B Implementation Details · p. 12
- Baseline comparisons use one white-box and four state-of-the-art black-box methods, with a 200-query limit for the methods.Additional settings specify datasets and embedding models for ablations, small LLMs and target models for paraphrasing attacks, and four Qwen2.5-7B derivatives for overhead evaluation.Source: B Implementation Details · p. 12
C Case Study of Closed-source Audit
In a closed-source audit case study, ZeroPrint assigns high fingerprint similarity to repeated queries of GPT-4o-mini and low similarity to three independent LLMs.
C Case Study of Closed-source Audit
- 0.943 fingerprint similarity for GPT-4o-mini contrasts with approximately 0.50 for Gemini-2.5-Flash-Lite, Claude-Haiku-4.5, and DeepSeek-V3.The study uses the same GPT-4o-mini API as source and derivative.Source: C Case Study of Closed-source Audit · p. 12
D Ethic Statement
ZeroPrint is presented as a defensive tool for copyright auditing, using public datasets and involving no human subjects; the authors report no major ethical issues.
D Ethic Statement
- ZeroPrint is designed for the defensive purpose of copyright auditing.The authors state that the method introduces no new threats or vulnerabilities.Source: D Ethic Statement · p. 12
- The work uses publicly available datasets such as HumanEval and does not involve human subjects or infringe on individual privacy.Source: D Ethic Statement · p. 12
- The authors conclude that the work raises no major ethical issues.Source: D Ethic Statement · p. 12