PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Model Fingerprinting Requires Rethinking Watermark Teachers

Jeongyeon Hwang, Anshul Nasery, Sewoong Oh, Jungseul Ok

arXiv:2610.04169v1cs.CRcs.AI

TL;DR

Watermark-distillation fingerprinting needs to verify model ownership behind black-box APIs, but existing utility evaluation understates open-ended text degradation. The paper analyzes sparse signal placement and proposes near-tie restriction, which improves detection–quality frontiers across models and schemes. Its results support tailoring watermark teachers to model verification rather than inheriting text-watermark objectives.

  • Problem

    A recent watermark-based fingerprinting protocol’s utility evaluation understates open-ended generation degradation, while weakening the watermark improves quality at the cost of detectability.

    Source: §1 · p. 1

  • Method

    The paper uses token-surprisal analysis to motivate near-tie restriction, limiting watermark bias to tokens close to the base model’s top prediction.

    Source: §1 · p. 2, §5.2 · p. 5

  • Results

    Near-tie improves detection–quality frontiers across models and schemes under deployment transformations, preserves higher text quality across query budgets, and further improves existing schemes when combined with them.

    Source: §1 · p. 2, §6.2 · p. 8, §7 · p. 10

  • Takeaways & Limitations

    Fingerprint verification can aggregate signal across queries, so watermark teachers can use sparser signals placed on tokens close to the model’s top prediction.

    Source: §1 · p. 2, §7 · p. 10

    The verification setting assumes the owner retains the base model and secret key, while suspect-model access is limited to queries and responses that may have undergone unknown transformations.

    Source: §3 · p. 4

Abstract

from arXiv · show

LLM fingerprinting via watermark distillation embeds a statistical watermark signal into model weights, enabling model owners to identify their models behind black-box APIs. Revisiting a recent protocol, we find that its utility evaluation understates text quality degradation in open-ended generation, favoring overly strong watermark teachers. Weakening the watermark improves text quality but sacrifices detectability. To move beyond this trade-off, we rethink whether text watermarks designed for verifying generated text are suitable distillation teachers for model fingerprinting. Such watermarks are typically designed to remain detectable from an individual output, limiting how sparse the watermark signal can be. In contrast, fingerprint verification can aggregate signal across queries, making sparser watermark signals viable. This raises a key question: where should the sparse signal be placed? We analyze signal placement through token surprisal and show that, even at comparable watermark strength, different placements can target tokens with different plausibility under the base model. This motivates near-tie restriction, which uses top-1-relative logit gaps to restrict the watermark bias to tokens close to the base model's top prediction. Across multiple models, near-tie improves detection--quality frontiers under deployment changes, preserves higher text quality across query budgets, and further improves existing watermarking schemes when combined with them.

1 INTRODUCTION

Black-box model fingerprinting addresses ownership verification when open-weight models are deployed behind APIs, but existing watermark teachers can substantially degrade open-ended generation quality. The paper motivates near-tie restriction and reports improved detection–quality frontiers across models and watermarking schemes.

1 INTRODUCTION

  • The ownership-verification problem grows as open-weight models are deployed behind black-box APIs without necessarily honoring restrictive licenses.The paper frames reliable model provenance verification as important for a healthy ecosystem.Source: §1 · p. 1
  • The evaluated protocol understates open-ended generation degradation, favoring strong watermark teachers; weakening the watermark improves quality but sacrifices detectability.More advanced watermarking schemes provide only limited gains as distillation teachers.Source: §1 · pp. 1–2
  • Fingerprint verification can aggregate evidence across queries, making sparser teacher signals viable than watermarks designed for detection from individual outputs.The paper analyzes signal placement through token surprisal and proposes restricting bias to green tokens near the model’s top prediction.Source: §1 · p. 2, Figure 1 · p. 2
  • Near-tie substantially improves detection–quality frontiers across multiple models and watermarking schemes under common deployment transformations.It also preserves higher generation quality across query budgets and further improves existing schemes when combined with them.Source: §1 · p. 2

2 RELATED WORK

The related work contrasts white-box fingerprinting, which requires weight access, with black-box approaches that verify ownership through queries and responses. It situates watermark-based fingerprinting among text watermarking and watermark distillation methods.

LLM fingerprinting

  • White-box fingerprinting provides robust verification but requires model-weight access, motivating black-box verification through queries and responses.Backdoor-based methods use secret trigger queries and exact-match responses, while watermark-based methods distill a watermark into model weights for statistical verification (Gloaguen et al., 2026).Source: §2 · p. 2
  • Backdoor-based methods can lose stealthiness through atypical query–response pairs, while exact-match verification can be fragile to deployment transformations.The passage cites Xu et al. (2024a), Nasery et al. (2025b), and Russinovich & Salem (2024) for backdoor-based approaches.Source: §2 · p. 2

LLM watermarking

  • The authors report that existing fingerprinting evaluation understates open-ended generation degradation and note that fingerprint verification aggregates evidence across responses.They use this distinction to motivate a sparser teacher intended to preserve text quality.Source: §2 · p. 3
  • Text watermarking embeds a hidden statistical signal in generated text, and watermark distillation has been extended to semantic-domain model fingerprinting.The related work describes KGW (Kirchenbauer et al., 2023), alternative schemes (Lee et al., 2024; Wang et al., 2025b; Wu et al., 2023; Liu et al., 2024), and watermark distillation (Gu et al., 2024; Gloaguen et al., 2026).Source: §2 · p. 3

3 PRELIMINARY

The preliminary section defines the next-token prediction and red–green watermarking setup, then describes watermark distillation and black-box fingerprint verification. Verification aggregates responses while removing duplicate h-grams before scoring.

3 PRELIMINARY

  • A language model predicts the next token from preceding tokens by producing logits that define a distribution through softmax, from which decoding samples.The sampled token depends on the decoding strategy.Source: §3 · p. 3
  • KGW partitions the vocabulary into secret-key-dependent green and red sets and adds bias δ to green-token logits before softmax.Its detector compares the generated green-token count with the null expectation γL and classifies a response as watermarked when z(x) exceeds threshold ρ.Source: §3 · p. 3
  • Watermark-based fingerprinting distills a semantically conditioned watermark teacher into model weights so the model emits a detectable signal without modified inference-time decoding.The training objective includes a regularizer to suppress watermarked generations outside the fingerprint domain.Source: §3 · p. 3
  • For black-box verification, the owner queries the fingerprint domain and removes duplicate h-grams across responses to avoid counting repeated watermark evidence multiple times.Retained tokens are evaluated in their original response contexts before the detector computes its statistic.Source: §3 · pp. 3–4
  • The threat model gives the owner the base model and secret key but limits access to a suspect model to queries and observed responses, potentially after unknown transformations.The listed transformations include sampling changes, quantization, pruning, and fine-tuning.Source: §3 · p. 4

4 REVISITING UTILITY EVALUATION OF LLM FINGERPRINTING

Reevaluating a prior fingerprinting setting on open-ended French generation reveals substantial quality degradation that benchmark accuracy and mean token entropy understate. Reducing watermark bias can mitigate that degradation, but weaker bias compromises detectability.

4 REVISITING UTILITY EVALUATION OF LLM FINGERPRINTING

  • Accuracy fell only from 0.64 to 0.61, while PPL rose from 4.63 to 28.27 and LLM-judge score fell from 7.76 to 3.73 in open-ended generation.The evaluation used 500 French-translated WritingPrompts examples for the Llama-3.1-8B-Instruct setting.Source: §4 · p. 4
  • The utility evaluation used accuracy-based tasks and mean token entropy, rather than open-ended generation quality and standard perplexity.The official implementation computes the reported PPL as mean token entropy.Source: §4 · p. 4
  • δ=2 lowers the PPL ratio while maintaining robust detectability, but still more than doubles base-model PPL; δ=1 falls below the detection threshold.The bias sweep compares δ values 1, 2, and 4 using PPL ratio and worst-case z-score under transformations.Source: §4 · p. 4
  • The bias-strength trade-off motivates fingerprinting schemes with a better detection–quality frontier.The figure compares detection–quality trade-offs across logit biases, with the upper-left region preferred.Source: Figure 2 · p. 4, §4 · p. 5

5 RETHINKING WATERMARKED TEACHERS FOR MODEL FINGERPRINTING

Section 5 reframes watermark distillation around fingerprinting’s ability to accumulate evidence across queries. It analyzes sparse signal placement by base-model plausibility and introduces near-tie restriction.

5.1 FINGERPRINT VERIFICATION ALLOWS SPARSE WATERMARK SIGNALS

  • Fingerprint verification can aggregate evidence across queries, so sparse teacher signals are viable and may reduce base-distribution distortion compared with dense green-token biasing.The paper links dense KGW teacher biases transferred through distillation to model degradation.Source: §5.1 · p. 5

5.2 WHERE SHOULD THE SPARSE SIGNAL BE PLACED?

  • For equal-mass subsets receiving the same bias, lower-average-surprisal green tokens provide equal expected watermark evidence and KL divergence but induce a smaller surprisal shift.The analysis therefore favors assigning the bias to green tokens the base model considers more plausible.Source: §5.2 · p. 6

5.3 OUR APPROACH: NEAR-TIE RESTRICTION

  • Near-tie restriction biases green tokens within a threshold of the top-1 logit, with τ calibrated from training-data top-1–top-2 gaps; smaller percentiles bias fewer tokens.A threshold tending to infinity recovers the original dense watermark.Source: §5.3 · p. 6
  • Verification re-scores responses with the retained base model to select near-tie positions; key-independent selection and h-gram deduplication preserve the null green-token rate γ for the standard detector.The scored positions retain their original within-response contexts.Source: §5.3 · p. 6
  • Near-tie restriction can further restrict SWEET and MorphMark’s biased green-token sets, complementing their entropy-based and cumulative-probability mechanisms.The combined variants are SWEET-NT and MorphMark-NT.Source: §5.3 · p. 6

6 EXPERIMENTS

Experiments across models and deployment settings show that KGW-NT improves detection–quality trade-offs and preserves text quality across query budgets. Ablations examine restriction strength, signal aggregation and placement, and combining near-tie with existing watermarks.

6.1 SETUP

  • Across four instruction-tuned models, the primary French fingerprint domain uses French Alpaca queries and French WritingPrompts for open-ended generation evaluation.The study also evaluates a medical fingerprint domain and OrangeSum, with improvements over baselines reported for both.Source: §6.1 · pp. 6–7

6.2 MAIN RESULTS

  • KGW-NT improves the detection–quality frontier across models and quality metrics, outperforming KGW-TK and suggesting logit-gap selection matters beyond sparsity alone.It achieves higher worst-case detectability at comparable quality or better quality; rank-based top-k selection can include tokens far from the top-1 prediction.Source: Figure 3 · p. 7, §6.2 · p. 8
  • KGW-NT achieves the lowest PPL ratio at each query budget while maintaining robust detectability.Larger budgets allow higher text quality at comparable detectability, and some 3B settings achieve PPL ratios below 1.Source: Figure 4 · p. 7, §6.2 · p. 8

6.1 SETUP

  • KGW-NT is compared with KGW, SWEET, MorphMark, and KGW-TK across five deployment transformations, using worst-case z-scores, PPL, and LLM-judge quality.Transformations include sampling, system prompts, quantization, pruning, and fine-tuning.Source: §6.1 · pp. 7–8

6.3 ABLATION STUDIES AND ANALYSIS

  • Moderate restriction percentiles q = 30–50 give the most favorable trade-off; q = 70 substantially increases the PPL ratio.Expanding the biased support generally improves detectability, while overly broad support sacrifices text quality.Source: Figure 5 · p. 8, §6.3 · p. 9
  • With q = 10 or q = 30, sparse signals remain below the detection threshold at 256 tokens but become detectable as evidence accumulates across more tokens.This supports using multi-response fingerprint verification for signals too sparse for detection from a single output.Source: Figure 5 · p. 8, §6.3 · p. 9
  • At matched green mass and KL, random placement produces greater surprisal shifts and larger student PPL ratios than near-tie placement.The comparison changes which tokens carry the signal while holding teacher strength constant.Source: Table 2 · p. 9, §6.3 · p. 9
  • Adding near-tie improves SWEET and MorphMark detection–quality frontiers toward lower PPL ratios while maintaining strong worst-case detection.The joint hyperparameter space was not exhaustively explored.Source: Figure 6 · p. 9, §6.3 · p. 9

7 CONCLUSION

The paper argues that watermark teachers for fingerprinting should be designed for model verification rather than inherit text-watermarking objectives. Near-tie restricts bias to tokens close to the model’s top prediction and can complement existing watermark mechanisms.

7 CONCLUSION

  • The recent protocol’s utility evaluation understates open-ended generation degradation, motivating watermark teachers designed for model verification rather than text-watermarking objectives.The conclusion says this degradation can favor overly strong teachers that harm generation quality for legitimate users.Source: §7 · p. 10

A EXTENDED RELATED WORK

  • Black-box fingerprinting either exploits existing model behavior or modifies the model; modification allows robustness to be considered in fingerprint design but can degrade utility.Existing-behavior methods avoid fingerprint-induced utility degradation but may be vulnerable to transformations.Source: A EXTENDED RELATED WORK · p. 14
  • Targeted methods search for distinctive query–response pairs but can use atypical, filterable queries, while untargeted methods use natural queries but can be vulnerable to fine-tuning.The passage also describes robustness extensions for targeted methods against model merging, fine-tuning, and system prompts.Source: A EXTENDED RELATED WORK · p. 14
  • Watermarking alternatives adjust where or how bias is applied, including high-entropy selection, adaptive strength, distribution-preserving reweighting, and sparse position selection.The related work also describes logit-balanced partitions, lexical redundancy, and Pareto optimization within a watermark class.Source: A EXTENDED RELATED WORK · p. 14
  • Near-tie uses top-1-relative logit gaps to restrict which tokens receive watermark bias, and its restriction is complementary to existing watermark mechanisms.The method can be combined with those mechanisms while preserving their original designs.Source: A EXTENDED RELATED WORK · p. 14

B TRAINING DETAILS FOR WATERMARK-BASED FINGERPRINTING

The training protocol distills the fingerprint-domain teacher while regularizing the model on separate data to preserve the base distribution outside that domain. The proof examines how selected token sets affect biased probability mass, divergence, and expected surprisal.

B TRAINING DETAILS FOR WATERMARK-BASED FINGERPRINTING

  • The objective combines in-domain fingerprint distillation with regularization on Dreg, whose one-sided total-variation term penalizes token-probability increases outside the fingerprint domain.The stated purpose of the regularization dataset is to prevent watermark leakage outside the domain.Source: B TRAINING DETAILS FOR WATERMARK-BASED… · p. 15
  • Training uses Lucie for Dfp and an equal Alpaca–OpenWebText mixture for Dreg, with 2,500 steps, batch size 64, and Adafactor at 2 × 10^-5.The setup uses 250 warmup steps, a cosine schedule, h = 1, γ = 0.25, and varies δ and method-specific parameters.Source: B TRAINING DETAILS FOR WATERMARK-BASED… · p. 15

C PROOF OF PROPOSITION 5.1

  • For selected token sets S with the same base probability mass m, the proof states that biased green-token mass and KL divergence depend on S only through m.It then compares their expected base-model surprisal under the biased distribution.Source: C PROOF OF PROPOSITION 5.1 · p. 16

D NULL CALIBRATION OF THE NEAR-TIE DETECTOR

The appendix establishes that near-tie restriction preserves the analytical KGW detector’s null green-token rate and reports empirical null calibration across restriction percentiles. It also specifies baseline sweeps and deployment transformations used in the experiments.

D NULL CALIBRATION OF THE NEAR-TIE DETECTOR

  • The standard KGW normalization is conservative under the stated dependence structure, and the analytical detector’s null green-token rate is preserved under near-tie restriction.The resampling check found null standard deviations close to one, with every 95% confidence interval containing 1 and estimates differing by less than 0.01 across q.Source: D NULL CALIBRATION OF THE NEAR-TIE… · p. 17
  • 17.30 to 12.90 (z-score) and 1.02 to 1.18 (PPL ratio) for q = 30 when the position-level gate is applied to Llama-3.1-8B-Instruct at δ = 4.The passage interprets these results as suggesting that green top-1 biases provide useful distillation signal.Source: D NULL CALIBRATION OF THE NEAR-TIE… · p. 17

E BASELINE HYPERPARAMETERS

  • For KGW, SWEET, KGW-TK, and KGW-NT, δ is swept over {1, 2, 4}; SWEET entropy and KGW-NT logit-gap thresholds use training-data percentiles.KGW-TK sweeps k over {2, 5, 10, 50}; MorphMark sweeps kexp and p0 while fixing γ = 0.5.Source: E BASELINE HYPERPARAMETERS · p. 18

F DETAILS OF DEPLOYMENT TRANSFORMATIONS

  • Table 5 reports mean ± standard deviation over five runs for each model and transformation; the system-prompt tests include acknowledgment, reasoning, and advertising prompts.The section points to Section 6.1 for the remaining transformations.Source: F DETAILS OF DEPLOYMENT TRANSFORMATIONS · p. 18
  • The fine-tuning transformation uses one epoch on the French WildChat subset, batch size 64, Adafactor, and a learning rate of 2 × 10^-5.Source: F DETAILS OF DEPLOYMENT TRANSFORMATIONS · p. 18

G IMPLEMENTATION DETAILS FOR NEAR-TIE COMBINATIONS

Across evaluated models, KGW-NT improves detection–quality trade-offs and achieves the lowest PPL ratio across query budgets. Near-tie also improves existing watermark schemes when combined with them.

G IMPLEMENTATION DETAILS FOR NEAR-TIE COMBINATIONS

  • Near-tie combines with an existing watermark by further restricting its biased green-token set to tokens meeting the near-tie criterion, while otherwise retaining the original mechanism.SWEET-NT also requires its entropy criterion for detection; MorphMark-NT uses the near-tie detector, whereas KGW-TK uses the standard KGW detector.Source: G IMPLEMENTATION DETAILS FOR NEAR-TIE… · p. 18
  • KGW-NT consistently improves the Pareto frontier across model families and text-quality metrics, with higher worst-case detectability at comparable quality or better quality at comparable detectability.Across models and query budgets, it also achieves the lowest PPL ratio among operating points meeting the detection threshold.Source: Figure 7 · p. 19, Figure 8 · p. 19
  • Near-tie improves SWEET and MorphMark LLM-judge trade-offs, while KGW-NT better preserves text quality than SWEET-NT and MorphMark-NT at robustly detectable settings.Source: Figure 9 · p. 19

H DETAILED EXPERIMENTAL RESULTS

KGW-NT consistently improves detection–quality trade-offs across evaluated models and metrics. It also maintains higher text quality across query budgets and performs well on OrangeSum and when combined with other watermark schemes.

H DETAILED EXPERIMENTAL RESULTS

  • Across four models, KGW-NT consistently improves the detection–quality trade-off, with LLM-judge results matching the PPL-ratio trend.For Gemma-2-9B, PPL improvement is less pronounced, but KGW-NT achieves stronger detectability at comparable text quality.Source: H DETAILED EXPERIMENTAL RESULTS · p. 19

I.1 EVALUATION ON AN ADDITIONAL DATASET

  • KGW-NT consistently improves detection–quality trade-offs on OrangeSum across models, with LLM-judge results matching the PPL-based evaluation.OrangeSum is a French news dataset.Source: Figure 10 · p. 20, §I.1 · p. 20

H DETAILED EXPERIMENTAL RESULTS

  • KGW-NT consistently achieves the lowest PPL ratio across query budgets, preserving higher text quality while maintaining robust detectability.The passage reports this pattern across evaluated models, including Gemma-2-9B.Source: H DETAILED EXPERIMENTAL RESULTS · p. 20
  • Near-tie variants of KGW, SWEET, and MorphMark improve text quality among robustly detectable configurations.LLM-judge evaluation shows the same improvement for SWEET and MorphMark as the PPL-based results.Source: H DETAILED EXPERIMENTAL RESULTS · p. 20

I.2 EVALUATION ON AN ADDITIONAL FINGERPRINT DOMAIN

The additional-domain evaluation reports improved detection–quality trade-offs in medicine, and component analysis finds gains from both teacher-side near-tie restriction and a near-tie detector. Further analyses characterize output diversity and entropy, compare temperature sweeps, and describe the text-quality judging rubric.

I.2 EVALUATION ON AN ADDITIONAL FINGERPRINT DOMAIN

  • KGW-NT improves the detection–quality trade-off in the medicine fingerprint domain under both PPL and LLM-judge evaluation.The evaluation uses health data for fingerprint distillation and Medicine wiki data for verification.Source: Figure 11 · p. 20, §I.2 · p. 20

I.3 EFFECT OF THE NEAR-TIE DETECTOR

  • KGW-NT outperforms KGW under the standard detector, indicating that restricting teacher bias to near-tie tokens improves the trade-off independently of the detector.Applying the near-tie detector to KGW improves detectability without changing its PPL ratio; applying it to KGW-NT adds detectability gains.Source: §I.3 · p. 21, Figure 12 · p. 21

I.4 ANALYSIS OF DIVERSITY AND OUTPUT ENTROPY

  • KGW-NT maintains distinct-n-gram scores close to the base model, with slightly higher self-BLEU and comparable or higher LLM-judge scores.For Llama-3.1-8B, KGW-TK and KGW-NT (q = 30, δ = 4) have similar output entropy (1.05 vs. 1.03) but different PPL ratios (1.20 vs. 1.02).Source: §I.4 · p. 21, Table 4 · p. 22

I.5 CAN TEMPERATURE TUNING RECOVER THE NEAR-TIE FRONTIER?

  • Figure 13 compares KGW frontiers across decoding-temperature sweeps with KGW-NT at T = 1.0.The supplied passage identifies the comparison but does not report its outcome.Source: Figure 13 · p. 21

J PROMPT FOR TEXT QUALITY JUDGMENT

  • The text-quality judge assesses linguistic degradation, including grammatical errors, disfluency, malformed words, and excessive repetition, rather than creativity.The French-language prompt scores generated text from 1 to 10 and excludes truncation itself as a quality defect.Source: J PROMPT FOR TEXT QUALITY JUDGMENT · p. 24

Related papers

Loading 2610.04169v1…