PaperlayerView exact revision on arXiv

Source-linked AI summary

RAFP: Identifying LLM Lineages via Rare-Region Fingerprints

Yun-Yun Tsai, Jia Hao Liang, Chuan Guo, Junfeng Yang, Laurens van der Maaten

arXiv:2505.12682v2cs.LG

TL;DR

Model ownership verification is difficult when fingerprinting methods are fragile under adaptation or require white-box access. RAFP uses rare-region prompt-response fingerprints and theoretical analysis of their stability under finetuning. Experiments report persistent fingerprints and stronger performance than prior baselines in black-box settings.

  • Problem

    Model ownership verification matters as LLMs are released under restricted licenses, but existing fingerprinting methods can be fragile under finetuning, invasive, or unsuitable for black-box access.

    Source: §1 · p. 1

  • Method

    RAFP constructs lineage-specific prompt-response fingerprints from rare regions, where finetuning updates are weak and gradients align weakly with common-region updates.

    Source: §1 · pp. 1–2

  • Results

    Across multiple LLM families and downstream adaptations, RAFP achieves strong fingerprint persistence and substantially outperforms prior fingerprinting baselines in black-box settings.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Rare-region fingerprints can support identification of same-lineage models across common adaptations while distinguishing unrelated models.

    Source: §1 · pp. 1–2

    Stronger adaptations such as distillation, reparameterization, or adversarial finetuning may weaken or erase fingerprints, while black-box verification also faces query costs and filtering.

    Source: §8 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly released under restricted licenses, creating a growing need for robust model ownership verification. Existing fingerprinting methods are often fragile under downstream finetuning, require invasive training modifications, or fail in black-box settings. We introduce RAFP, a robust framework for identifying LLM lineages via rare-region fingerprints. Our key insight is that downstream finetuning primarily updates common high-density language behaviors, while low-probability prompt regions receive weak optimization signal and limited gradient alignment under finetuned distribution. As a result, rare prompt-response behaviors remain stable across common model adaptations. RAFP is non-invasive, constructing fingerprints via discrete gradient-based optimization over rare prompts without modifying model weights. We provide a theoretical analysis showing that the likelihood change of rare-region fingerprints under finetuning remains bounded. Experiments across four LLM families and multiple downstream adaptations, including supervised finetuning, LoRA, quantization, prompt-template variation, and decoding changes, show that RAFP achieves strong fingerprint persistence and substantially outperforms prior fingerprinting baselines in black-box settings.

1 Introduction

RAFP addresses the challenge of identifying stolen LLMs when model changes make similarity-based detection unreliable and access is limited to an API. It fingerprints rare prompt-response behaviors, which the paper argues remain stable under common adaptations, and evaluates the approach across models and attacks.

1 Introduction

  • Model theft can cause significant economic losses, while similarity-based detection is vulnerable to finetuning, pruning, and quantization and often cannot inspect API-only models.The paper motivates methods that are robust to common modifications and work in black-box settings.Source: §1 · p. 1
  • RAFP fingerprints model lineages with prompt-response pairs from rare regions, where downstream finetuning has weak optimization signal and gradient alignment.The paper argues this leaves rare-region behaviors largely invariant, helping same-lineage models retain the response while unrelated models do not.Source: §1 · p. 1
  • Owners can commit cryptographic hashes before deployment and withhold fingerprint queries until ownership verification, reducing exposure to filtering.The suspect model is queried through a black-box protocol.Source: §1 · pp. 1–2
  • For rare-region prompts, the paper proves that finetuning-induced log-likelihood change remains bounded because rare-prompt updates are small and common-region updates align weakly with rare-prompt gradients.The analysis formalizes the change as Δ(x, y) = log pθ′(y | x) − log pθ(y | x).Source: §1 · p. 2
  • Experiments across multiple LLM families report stable, discriminative fingerprints under finetuning, system-prompt variation, and quantization, and robustness to filtering and race attacks.The paper presents these findings as supporting model identification in black-box scenarios.Source: §1 · p. 2

2 Related Work

Prior LLM IP-protection work includes watermarking and fingerprinting approaches, with different trade-offs in invasiveness, access, and resilience to adaptation.

2 Related Work

3 Preliminaries

The paper formulates lineage identification as black-box verification of whether a suspect model derives from a base model. It sets out the adversary assumptions and desired properties for such a scheme.

3 Preliminaries

  • The adversary is assumed unable to retrain from scratch or tamper with pretraining, and limited to value-preserving post-training changes such as SFT, LoRA, prompt engineering, and quantization.Targeted input filtering or output rewriting is also considered, with a possible trade-off against normal generation quality.Source: §3 · pp. 2–3
  • Table 1 compares fingerprinting methods against six desired properties, including IF-SFT and GCG-QA as example methods.The caption identifies IF-SFT as instructional-finetuning-based and GCG-QA as using GCG on question-answering pairs.Source: Table 1 · p. 3, §3 · p. 3
  • The verification task is to decide whether a suspect model derives from a base model using black-box access, with fingerprints represented as prompt-response pairs.The scheme generates a fingerprint set and checks whether the suspect model exhibits its responses.Source: §3 · p. 3
  • Desired properties include effective identification, robustness across lineage adaptations, uniqueness across lineages, and difficulty for attackers to discover or reproduce fingerprints.The passages refer to effectiveness, robustness, uniqueness, and attack difficulty as scheme properties.Source: §3 · p. 3
  • RAFP generates fingerprints by optimizing unlikely token sequences, greedily decoding their responses, and searching across adapted model variants; verification checks TPR on the suspect model.The figure describes rare prompt-response pairs as difficult for an adversary to find and unlikely to be affected by common changes.Source: Figure 1 · p. 3
  • The scheme should support verification by querying a model without accessing its weights and should not degrade model quality.Source: §3 · p. 3

4 Robust Fingerprinting in RAFP

RAFP identifies model lineages using prompt-response fingerprints drawn from low-density regions, where downstream adaptation provides weaker optimization signal. The section describes fingerprint generation and verification, practical robustness strategies, and a theoretical account of stability under finetuning.

4 Robust Fingerprinting in RAFP

  • RAFP has two stages: it searches for rare prompts that induce stable responses, then checks whether a suspect model preserves those fingerprint behaviors.The approach targets prompts from low-density regions of the pretrained model distribution.Source: §4 · p. 3

4.1 Fingerprint Generation

  • Fingerprints are generated without changing model weights by finding rare prompts that reliably elicit the same response under decoding.The procedure initializes prompts with random and bottom-k sampling, generates a response greedily, then uses GCG to raise its likelihood.Source: §4.1 · p. 4

4.2 Ownership Verification

  • Ownership verification probes a suspect model with unrevealed prompts, checks for the committed responses, and tests unrelated lineages to reduce accidental matches.The developer first publishes cryptographic commitments to fingerprints, then can reveal a committed fingerprint to verify ownership.Source: §4.2 · p. 4

4.3 Practical Optimization

  • Joint optimization spans adapted models and system prompts; 20 successful independent GCG trials are required, and the example retains its response across prompt changes.The multi-trial strategy addresses brittle local optima; the figure illustrates a fingerprint response preserved under a system-prompt change.Source: §4.3 · p. 4, Figure 2 · p. 4

4.4 Stability under Finetuning

  • The rare-prompt region is defined by a low rarity score under the pretrained model, while its downstream probability mass measures overlap with finetuning prompts.Lower rarity scores indicate prompts less likely under the pretrained model; ρ(τ) measures how often downstream data overlaps with that region.Source: §4.4 · p. 5
  • Under scarcity and bounded-gradient assumptions, the lemma bounds the likelihood change after one downstream gradient update; rare prompts and limited gradient alignment explain the stability.The interpretation attributes stability to low downstream probability mass for rare prompts and limited alignment with most finetuning directions.Source: §4.4 · pp. 4–6

5 Experiments

Experiments test RAFP across four pretrained LLMs and multiple adaptations, reporting strong base-model effectiveness, finetuning persistence, and black-box uniqueness.

5.1 Experimental Setup

  • The evaluation covers Llama 2 7B and 13B, Llama 3 8B, and Mistral 7B, with SFT on three instruction and two conversational datasets.The SFT recipe trains for three epochs on each dataset.Source: §5.1 · p. 6

5.2 Experimentsal Results

  • RAFP achieves 100% TPR across four base models and 92∼100% across finetuned settings, improving TPR over standard GCG by 37∼80%.The comparison also reports 10–35% TPR degradation for IF-SFT and IF-Emb after finetuning.Source: Table 2 · p. 7, §5.2 · p. 7
  • Increasing optimization tasks from zero to two consistently improves TPR across all four models under both SFT and LoRA.Table 3 reports identification TPR across four pretrained models and 10 fingerprints.Source: Table 3 · p. 7, §5.2 · p. 7
  • In black-box evaluation, RAFP fingerprints all relevant Llama 2 7B derivatives while rejecting unrelated models at 0% FPR.The evaluation uses repository-recommended prompt templates and default decoding configurations.Source: §5.2 · p. 7

6 Ablation Studies

Ablations assess RAFP under perplexity filtering, prompt-template changes, decoding temperatures, and black-box uniqueness tests.

Evaluation on Uniqueness

  • Table 4 evaluates TPR on nine relevant and FNR on five unrelated Hugging Face models, marking successful fingerprinting or rejection with checkmarks.The caption specifies 10 QA pairs for the GCG-QA baseline.Source: Table 4 · p. 8

Robustness against Perplexity Filtering

  • Perplexity-aware optimization substantially reduces rejection rates while retaining moderate fingerprint effectiveness on base and SFT-adapted models.The joint objective adds prompt perplexity loss; λ = 0.1 lowers rejection without severely degrading TPR.Source: §6 · p. 8

Analysis of Prompt Templates

  • RAFP maintains 70–100% TPR across base and finetuned models under prompt-template variations, with the smallest TPR drop among the compared methods.The reported TPR degradation is 10–30%.Source: Table 6 · p. 8, §6 · p. 8

Analysis of Generation Hyperparameters

  • RAFP maintains nearly 100% TPR from temperature 0.1 to 0.7, with slight degradation above 0.8, while GCG-QA degrades from 0.3.Figure 3 evaluates temperatures ranging from 0 to 0.9.Source: Figure 3 · p. 8, §6 · p. 8

7 Conclusion

The conclusion presents rare-region fingerprints as a way to identify LLM lineages that remain stable under common adaptations. The authors pair a theoretical account of this stability with empirical results across model families and adaptations.

7 Conclusion

  • RAFP identifies LLM lineages using rare-region fingerprints whose prompt-response behaviors remain stable across model adaptations.The authors attribute this stability to rare prompt regions receiving weak optimization signal during downstream finetuning.Source: §7 · pp. 8–9
  • The paper provides theoretical analysis of rare-region fingerprint invariance under finetuning and reports persistence across multiple LLM families and downstream adaptations.The authors identify stronger adaptive adversaries and web-scale model training as directions for future work.Source: §7 · p. 9

8 Limitation •

RAFP has boundaries under strong adaptations and practical black-box use. Stronger changes may weaken fingerprints, while API access and filtering can constrain verification.

Vulnerability to Strong Adaptations

  • Distillation, reparameterization, and adversarial finetuning may weaken or erase fingerprints, although these adaptations are costly.The passage also identifies limited or costly API access and filtering or post-processing as practical challenges to black-box verification.Source: §8 · p. 9

Black-Box Constraints and Practicality

  • Filtering or post-processing can suppress fingerprints without degrading performance, posing a challenge for black-box verification.The passage identifies filtering and post-processing as challenges alongside limited or costly API access.Source: §8 · p. 9
  • Black-box verification requires querying the suspect model, and API access may be limited or costly.These constraints are stated alongside the risk of filtering or post-processing.Source: §8 · p. 9

9 Ethical Consideration

RAFP is presented as a way to track the origin of models involved in generating text, with potential benefits for transparency and attribution. The paper also raises privacy and governance concerns around identifying downstream model origins.

9 Ethical Consideration

  • Tracking the origin of models involved in generating particular texts may enhance transparency, aid attribution, and deter LLM misuse.The paper frames these as potential benefits of model lineage identification.Source: §9 · p. 9
  • Identifying downstream model origins may enable surveillance or monitoring that limits public-model use or suppresses free speech.The paper says these are risks, not inevitable outcomes.Source: §9 · p. 9
  • Future societal consequences are said to depend on regulatory and governance frameworks and clear policies for acceptable LLM uses.The paper also calls for recognition of which forms of fingerprinting can be used for model verification.Source: §9 · p. 9

A.1 Detailed Results of RAFP on Downstream Tuning (DT)

The appendix reports robustness, security, and harmlessness analyses of RAFP, including multi-stage tuning, quantization, fingerprint attacks, and the effects of fingerprint learning on model performance.

A.3 Robustness Analysis on Multi-Stage Fingerprint

  • After three finetuning stages, both multi-learning methods maintained 100% TPR on the base and finetuned models.The evaluation sequentially finetuned Llama 2 7B on three tasks; fingerprints were learned on one or two other, non-overlapping subtasks.Source: §A.3 · p. 13, Table 10 · p. 13

A.4 Robustness Analysis on Variant of Quantization

  • With 16-bit and 8-bit quantization, RAFP maintains high TPR while the MMLU score drops slightly; at 4-bit, TPR drops significantly and the MMLU score drops correspondingly.The authors say the 4-bit result severely decreases the model’s economic value.Source: §A.4 · p. 13

A.5 Dis-similarlity of bottom-k tokens

  • Different model lineages had extremely small bottom-k token-set similarity, and the calculated probability of randomly generating the same fingerprint was 1.95e-30.The Jaccard comparison used each base model’s last 2k tokens while holding supported languages constant.Source: §A.5 · pp. 13–14, Table 12 · p. 14

A.6 Front-running Attack Analysis

  • As fingerprint length increases, a front-running attacker needs more training samples to reach 100% TPR, though the required number remains relatively low at length 54.The authors suggest longer fingerprints and strong data deduplication to reduce this attack risk.Source: Figure 5 · p. 14, §A.6 · p. 14

A.7 Pre- and Post-Filtering Attack Analysis

  • Post-filtering with system prompts preserved high TPR, but perplexity differed between normal and fingerprint prompts, making perplexity-based fingerprint removal possible.The appendix also considers pre-filtering through a predefined prompt that filters nonsensical responses.Source: §A.7 · pp. 14–15, Figure 6 · p. 15, Figure 7 · p. 15

A.8 Unforgeable Property (Secrecy)

  • For random query-response pairs, the authors calculate a 1.95e-30 probability of matching a target response, using D = 2,000 and response length 9.The calculation assumes one valid response per query and a sufficiently random token mapping.Source: §A.8 · pp. 15–16
  • With D = 2,000, prompt length 32, and 10 fingerprints, 1,000 fake pairs yield a 4.68e-1027 erasure probability; a 1% probability would require 2.71e105 pairs.The passage concludes that completely erasing the fingerprints would require an astronomically large data volume.Source: §A.8 · p. 16

A.9 Harmless Property

  • Fingerprint learning with IF-SFT lowered MMLU scores on Llama 2 7B, Llama 2 13B, and Llama 3 8B, and learning rate also affected performance.The downstream-tuning tables report five models across SFT and LoRA variants; Table 9 presents modified system prompts, while the cited passages provide no cell values.Source: §A.1 · p. 12, §A.2 · p. 12, §A.9 · p. 16
Loading 2505.12682v2…