PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond QA Matching: Perturbation-Response Fingerprinting via Probability Distributions for Large Language Models

Jichao Zeng, Yanli Chen, Hanzhou Wu

arXiv:2609.06330v1cs.CR

TL;DR

Fine-grained provenance is difficult when LLM checkpoints are transformed and hard-answer comparisons lose probability information. BReF fingerprints models by comparing perturbation-induced probability responses, retrieving documented parents in all 22 benchmark cases while controls show pooled separation does not ensure equally strong candidate ranking.

  • Problem

    Because LLM checkpoints are often transformed, fine-grained provenance is difficult to determine from model outputs and their probability distributions.

    Source: §1 · p. 1

  • Method

    BReF selects 25 jointly responsive probes using response magnitude, then compares their perturbation log-ratio response directions with global cosine similarity.

    Source: §1 · p. 1, §4.2 · p. 4

  • Results

    BReF retrieved the documented parent at rank 1 for 22/22 suspects (MRR 1.0000; DP–DF AUC 1.0000), but strong pooled separation did not ensure equally strong candidate ranking.

    Source: §6.1 · p. 6, §8 · p. 10

  • Takeaways & Limitations

    Perturbation-response behavior provides a useful signal for fine-grained model provenance analysis, while same-family discrimination and answer-order sensitivity remain challenges.

    Source: §9 · p. 10

    BReF is robust to tested variations but not invariant to answer ordering or probability calibration.

    Source: §8 · p. 10

Abstract

from arXiv · show

Large language models are often instruction-tuned, specialized, quantized, or otherwise transformed, making fine-grained provenance difficult. In this paper, we introduce BReF, a training-free fingerprint that compares how probability distributions over four answer-option labels A/B/C/D move under controlled textual perturbations. For each pair of models, BReF selects 25 jointly responsive probes and compares their perturbation log-ratio (PLR) response directions by global cosine similarity. On a unified benchmark with 34 checkpoints, 22 documented direct-parent relations, and 411 suspect-candidate pairs, BReF retrieves the documented parent in 22/22 cases (MRR=1.0000), with DP-DF AUC 1.0000. Same-family discrimination is harder (DP-SF AUC 0.8969), and paired tests show a significant exact-retrieval gain over a magnitude-only Top-25 control. Together with static, random-probe, permutation, calibration, and transformation-level controls, the results show that strong pooled separation does not guarantee correct parent ranking among closely related checkpoints, verifying the superiority of our work.

1 Introduction

Fine-grained provenance is difficult because deployed LLM checkpoints are often transformed, while hard-answer matching discards probability information. BReF uses pair-specific perturbation responses to fingerprint models and is evaluated on a role-constrained direct-parent retrieval benchmark.

1 Introduction

  • Deployed LLM checkpoints are often instruction-tuned, specialized, quantized, or otherwise post-trained, making fine-grained identification difficult from outputs and probability distributions.Hard answer agreement also discards probability information when models choose the same option with different uncertainty.Source: §1 · p. 1
  • BReF selects 25 jointly responsive probes by response magnitude and compares their perturbation log-ratio directions using global cosine similarity.The probes are selected for each model pair.Source: §1 · p. 1, Figure 1 · p. 2
  • The benchmark includes 34 checkpoints, 22 documented direct-parent relations, a shared candidate registry, and 411 suspect–candidate pairs.The paper also lists controls for probabilities, probe selection and counts, option permutations, calibration, and tokenization.Source: §1 · p. 2
  • BReF achieves statistically significant gains over reproduced fingerprinting methods and a matched QA-agreement baseline on the direct-parent retrieval benchmark.The comparisons use the same candidate registry and apply multiple-comparison correction.Source: §1 · p. 2

2 Related Work

Prior fingerprinting work uses behavioral, boundary, representation, and parameter-based signals under different access assumptions. BReF instead compares perturbation-induced option-probability responses and evaluates semantic option permutations as a robustness control.

2 Related Work

  • Boundary and output-structure methods fingerprint decision boundaries, output correlations, or samples near boundary intersections, extending beyond hard-label agreement.BReF differs by using perturbation-induced LLM option-probability responses and pair-specific probe selection.Source: §2 · p. 2
  • LLM fingerprinting spans tuning-based, parameter-based, inference-time, and output-distribution approaches; BReF requires neither model modification nor hidden-state access, but assumes four designated option probabilities.The passage names examples including Instructional Fingerprinting, HuRef, LLMmap, REEF (Zhang et al., 2025), and MET.Source: §2 · p. 2
  • BReF fixes a common four-label probability protocol and tests semantic option permutations as an end-to-end robustness control.Table 1 is described as comparing methods’ signals and objectives; its caption defines “Option prob.” as direct A/B/C/D probabilities or sufficient option log-probabilities.Source: §2 · p. 2, Table 1 · p. 3

3 Problem Formulation

The protocol observes four answer-option probabilities for each multiple-choice probe and scores model pairs behaviorally. Direct-parent retrieval ranks a shared candidate registry against documented provenance roles, but the score is not proof of lineage or ownership.

3 Problem Formulation

  • The observation protocol uses a four-option probability vector for each probe, or normalizes the four designated option log-probabilities when those are available.The four labels A, B, C, and D denote answer options, not model identifiers.Source: §3 · p. 3
  • Interfaces that provide fewer than the four designated option values are outside the current protocol.The same prompt construction and answer-label convention apply across evaluated checkpoints.Source: §3 · p. 3
  • BReF scores behavioral similarity without family or lineage labels; retrieval is correct when the documented direct parent ranks highest among the registry candidates.Benchmark provenance defines suspect and candidate roles, while BReF supplies the scores used to rank the registry.Source: §3 · p. 3, Figure 2 · p. 3
  • A BReF similarity score is behavioral evidence for ranking, not standalone proof of causal lineage or legal ownership.Figure 2 depicts provenance-defined candidate roles and BReF ranking scores; its numerical scores are illustrative.Source: §3 · p. 3, Figure 2 · p. 3

4 BReF Method

BReF applies fixed textual perturbations, represents each probe through option-probability changes, selects probes responsive in both models, and compares response directions. Its cached comparison protocol still requires probability evaluations for the full probe and perturbation set.

4 BReF Method

  • BReF uses 13 fixed edits grouped schematically as Noise, Format, Swap, and Rewrite; the exact operations are listed in Appendix A.Figure 3 summarizes the scoring pipeline.Source: §4 · p. 3, Figure 3 · p. 4

4.1 Perturbation-response representation

  • The representation uses 971 four-option probes under an unperturbed baseline and 13 fixed perturbations, producing a 52-dimensional clipped PLR response per probe.PLR coordinates are response-relative log probability changes: positive values indicate an increase under perturbation and negative values a decrease.Source: §4.1 · pp. 3–4

4.2 Pair-specific perturbation-sensitive probe selection

  • For each model pair, the geometric mean of response magnitudes selects the 25 probes on which both models respond most strongly.Selection uses norms only, not response direction, family labels, or provenance labels; different candidate comparisons can select different subsets for the same suspect.Source: §4.2 · p. 4

4.3 Global response-direction similarity

  • After concatenating the selected responses in the same probe order, BReF uses global cosine similarity so response direction determines similarity.Magnitude determines where to compare; signed response-vector direction determines how similarly the models respond.Source: §4.3 · p. 4

4.4 Query and comparison cost

  • Selecting Top-25 probes after response extraction does not reduce the evaluation budget: each checkpoint requires 13,594 probability-returning inputs.Once cached, checkpoints can be compared without further inference; cache construction takes 6.56 minutes per model on one NVIDIA RTX A5000 GPU with 24 GB memory.Source: §4.4 · pp. 4–5

5 Experimental Setup

The experimental setup evaluates BReF on a unified direct-parent benchmark, defines retrieval metrics, and compares it with reproduced methods and a QA-agreement baseline. The benchmark also documents provenance evidence and specifies how its probes were constructed.

5.1 Unified direct-parent benchmark

  • The benchmark covers 34 open-weight checkpoints, 22 documented direct-parent edges, and 411 suspect–candidate pairs, with candidates spanning family and lineage relationships.Candidate pools include siblings, ancestors, descendants, and cross-family checkpoints; the 22 parent labels were independently audited against public provenance evidence.Source: §5.1 · p. 5
  • The 22 parent edges have public support: 15 receive Grade A evidence, six Grade B, and one conservative Grade C; repository revision hashes were not recorded at download.The evidence grades organize documentation strength and are not an external certification standard.Source: §5.1 · p. 5
  • The 971 probes come from a cleaned 1,014-item pool built from QQP, MRPC, ANLI, and IFEval using predefined A/B/C/D conversion templates.Verbal options are fixed during probe construction rather than taken from source datasets or generated by evaluated models.Source: §5.1 · p. 5

5.2 Metrics

  • Top-1, Top-3, MRR, and ROC-AUC support scale-insensitive cross-method comparisons; BReF also reports DP–DF separation and parent-retrieval margins with suspect-clustered uncertainty.Raw DP–DF gaps are interpreted only within a method because reproduced methods use different score semantics and scales.Source: §5.2 · p. 5

5.3 Baselines

  • Baselines reproduce REEF (Zhang et al., 2025), LLMmap, MET, and ZeroPrint on the same 22-case protocol.A QA-Agreement baseline uses A/B/C/D argmax agreement on the same probes and clean13 perturbations; comparisons assess retrieval performance, not equal-cost efficiency.Source: §5.3 · p. 6

6 Results

BReF ranks every documented parent first in the benchmark, with paired tests supporting gains over prior methods and magnitude-only controls. Results also show that close within-family ranking is harder and that probe coverage and transformations affect the evaluation.

6.2 Statistical reliability and paired significance

  • Against REEF, ∆MRR is 0.1829 (95% CI [0.0682, 0.3052]); all five Top-1 tests remain significant after Holm correction.Suspect-clustered intervals support the reliability estimates; family-cluster sensitivity checks remain above zero, with only six families available.Source: §6.2 · p. 6, Table 3 · p. 7

6.3 Coarse separation versus exact-parent ranking

  • DP–SF AUC is 0.8969 versus DP–DF AUC 1.0000; high pooled separation can coexist with retrieval misses, while pair-specific Top-25 reaches 22/22.Magnitude-only and all-probe controls can have high DP–DF AUC yet lower Top-1; QA-Agreement reaches 14/22 using the same probes and perturbations but only hard decisions.Source: §6.3 · pp. 6–7, Figure 4 · p. 7

6.1 Main direct-parent retrieval results

  • 22/22 suspects rank the documented parent first (MRR 1.0000; DP–DF AUC 1.0000), exceeding REEF's MRR 0.8171.Figure 4 visualizes the pair-score structure behind these retrieval results.Source: §6.1 · p. 6, Table 2 · p. 7

6.4 Energy-only controls and the role of response direction

  • 22/22 versus 14/22 Top-1 and MRR 1.0000 versus 0.7551 favor BReF over same-selector magnitude similarity.The paired Top-1 difference remains significant after Holm correction; joint-strength controls also perform poorly on the same pair-specific probes.Source: §6.4 · p. 8, Table 5 · p. 8

6.5 Probe-count and perturbation-coverage sensitivity

  • K = 25 is the smallest tested probe count with 22/22 Top-1; clean13 yields 22/22 versus 15.6/22 at L = 3.Across five nested perturbation orders, the mean rises to 21.0/22 at L = 12; clean13 has zero Top-1 variance.Source: §6.5 · p. 8, Figure 5 · p. 9

6.6 Semantic option permutation and calibration robustness

  • Across P1–P3, Top-1 is 20/22, 18/22, and 20/22; under asymmetric calibration, Top-1 ranges from 21/22 to 22/22.Permutation-related ranking degradations occur only in Qwen, while non-Qwen families remain perfect; calibrated DP–DF AUC is 1.0000 across all 25 settings.Source: §6.6 · pp. 8–10, Table 6 · p. 9

6.7 Transformation-level coverage and hard cases

  • BReF ranks the documented parent first for all 22 edges across four transformation groups, with MRR 1.000 throughout.The groups are chat/alignment, code/math/task fine-tuning, instruction tuning, and quantization.Source: §6.7 · p. 10

7 Protocol Audits and Reproducibility

The reproducibility audit reports consistent probability extraction across the evaluated checkpoints and repeat runs.

7 Protocol Audits and Reproducibility

  • All 34 tokenizers map unspaced A/B/C/D to distinct single tokens, and repeated Falcon runs produce bitwise-identical probability outputs.The maximum absolute difference is 0 and Pearson correlation is 1.Source: §7 · p. 10

8 Discussion and Limitations

The controls distinguish pooled relation separation from candidate ranking: strong separation need not produce equally strong rankings. Directional PLR comparison outperforms magnitude-only comparison statistically, while robustness and protocol scope remain qualified.

8 Discussion and Limitations

  • Strong pooled relation separation does not necessarily translate into equally strong candidate ranking.Static probabilities, all-probe PLR responses, and random probe subsets can preserve relation-level discrimination while producing different ranking behavior.Source: §8 · p. 10
  • Directional PLR comparison shows a clearer statistically supported difference from magnitude-only comparison than pair-specific probe selection does with the current suspect count.The incremental effect of pair-specific selection is more modest.Source: §8 · p. 10
  • Semantic option permutations affect several Qwen-family rankings, while moderate probability rescaling causes smaller changes.BReF is therefore not invariant to answer ordering or probability calibration.Source: §8 · p. 10
  • The protocol requires complete four-option probabilities or sufficient option log-probabilities, and the benchmark does not address ambiguous multi-parent merges.Source: §8 · p. 10

9 Conclusion

BReF uses perturbation-induced changes in four-option probabilities, with magnitude guiding probe selection and direction guiding similarity. The conclusion presents this geometry as a useful provenance signal, while the disclosed probe pool and protocol specify how the evaluation inputs were built.

9 Conclusion

  • Perturbation-response geometry provides information beyond hard answer agreement and response magnitude alone, while same-family discrimination and answer-order sensitivity remain challenges.The conclusion supports perturbation-response behavior as a useful signal for fine-grained model provenance analysis.Source: §9 · p. 10

A Probe Pool and Perturbation Disclosure

  • The 1,014-item candidate pool draws on QQP, MRPC, ANLI, and IFEval, with fixed conversion templates and predefined verbal options introduced during probe construction.The verbal options come from a seed library rather than the source datasets or evaluated models.Source: A Probe Pool and Perturbation Disclosure · p. 13
  • Text-only quality control and template deduplication reduce the pool to 971 probes, with construction independent of current benchmark performance and labels.Deduplication uses a fixed lexical Jaccard threshold of 0.92.Source: A Probe Pool and Perturbation Disclosure · p. 13
  • All 971 probes have complete A/B/C/D options, but the metadata does not establish balanced gold-answer positions; scoring instead uses probability changes, with semantic permutations as a control.Source: A Probe Pool and Perturbation Disclosure · p. 13
  • The clean13 perturbation bank was fixed before sensitivity analyses, and the executable transformation functions are released with the artifact.Source: A Probe Pool and Perturbation Disclosure · p. 13
  • The protocol uses unspaced A/B/C/D labels, which all 34 evaluated tokenizers represent as distinct single tokens; scoring requires the four option probabilities.Token IDs are needed only for the local extraction audit.Source: A Probe Pool and Perturbation Disclosure · p. 13

B Unified Direct-Parent Benchmark

The benchmark uses a shared candidate registry and publicly documented parent relations, with same-family hard negatives retained. Baseline protocols preserve each method’s original signal while ranking the same candidates.

B Unified Direct-Parent Benchmark

  • The benchmark comprises 34 checkpoints, 22 suspects and documented direct-parent edges, 19 candidate parents, and 411 valid suspect–candidate pairs.Exact self-candidates are removed where applicable, while same-family hard negatives remain.Source: B Unified Direct-Parent Benchmark · p. 13
  • All 22 parent edges have public provenance evidence, and the perfect exact-parent retrieval result is not driven by the single lower-confidence edge.The evidence audit assigns 15 edges Grade A, six Grade B, and one Grade C; these are author-defined categories, not external certification.Source: B Unified Direct-Parent Benchmark · pp. 13–14

C Baseline Reproduction Details

  • Reproduced baselines use the same suspects and candidate registry, preserve each method’s original fingerprinting signal, and rank candidates by its final score.Reported budgets provide transparency but do not imply equal computational cost.Source: C Baseline Reproduction Details · p. 14

C.1 REEF

The section reproduces five baselines on a shared direct-parent benchmark, reporting their retrieval performance and examining sensitivity to probe and perturbation counts.

C.1 REEF

  • REEF retrieves 15/22 parents at Top-1, with Top-3 21/22, MRR 0.8171, DP–DF AUC 0.9679, and DP–All AUC 0.9507.It extracts last-token activations at block 18 from 200 statements, standardizes features, and ranks candidates with linear CKA.Source: §C.1 · pp. 14–15

C.2 LLMmap

  • LLMmap retrieves 5/22 parents at Top-1, with Top-3 11/22, MRR 0.4104, DP–DF AUC 0.7027, and DP–All AUC 0.6938.It encodes eight query-response traces per checkpoint into fingerprints and ranks candidates by cosine distance.Source: §C.2 · p. 15

C.3 Model Equality Testing

  • MET retrieves 14/22 parents at Top-1, with Top-3 15/22, MRR 0.6962, DP–DF AUC 0.7787, and DP–All AUC 0.7752.Its reproduced protocol uses 25 multilingual Wikipedia prompts and 250 completions per checkpoint, ranking candidates by MMD distance.Source: §C.3 · pp. 15–16

C.4 ZeroPrint

  • ZeroPrint retrieves 7/22 parents at Top-1, with Top-3 9/22, MRR 0.4234, DP–DF AUC 0.7142, and DP–All AUC 0.6933.It estimates query-level fingerprints from perturbation-induced changes in input and output embeddings, then compares them by Pearson correlation.Source: §C.4 · p. 16

C.5 QA-Agreement

  • QA-Agreement retrieves 14/22 parents at Top-1, with Top-3 16/22, MRR 0.720231, DP–DF AUC 0.856690, and DP–All AUC 0.835359.Its score uses hard decisions under perturbed conditions, without PLR or BReF’s pair-specific selector.Source: §C.5 · p. 17

D.1 K-sensitivity details

  • Figure 5(a) tracks ranking as the selected-probe count varies; the neighboring K = 26 and K = 27 settings also have perfect retrieval.Table 13 lists selected reporting points from the K-sensitivity analysis.Source: §D.1 · p. 17, Table 13 · p. 17

D.2 Perturbation-count stability

  • At L = 13, nested perturbation orders reach Top-1 22.0 ± 0 /22 and MRR 1.0000 ± 0, compared with 15.6 ± 3.29 /22 and 0.8081 ± 0.1055 at L = 3.The analysis averages results across five nested random orders.Source: §D.2 · p. 17

D.3 Selector and static controls

The controls probe selector and score behavior, option-order and calibration robustness, statistical uncertainty, reproducibility, and ethical boundaries.

D.3 Selector and static controls

  • Across 50 seeds, Random25 reaches DP–All AUC 0.9683 ± 0.0196 but Top-1 averages 18.04 ± 1.31/22, illustrating that pooled separation does not ensure stable exact-parent ranking.The controls distinguish joint response strength, magnitude similarity, and signed response-vector direction.Source: §D.3 · pp. 17–18

E.1 Semantic option permutation

  • Fresh option permutations leave evaluated families at perfect Top-1 except Qwen, where the principal rank changes involve four Qwen2.5 checkpoints.The reported changes involve Qwen2.5-7B-Instruct, Qwen2.5-Coder-7B, Qwen2.5-Math-7B, and Qwen2.5-Math-7B-Instruct.Source: §E.1 · p. 18, Table 15 · p. 18

E.2 Probability calibration / temperature robustness

  • Across the asymmetric 5 × 5 temperature grid, DP–DF AUC remains 1.000000 while DP–SF AUC ranges 0.896406–0.905391.Top-1 ranges from 21–22/22, and all settings retain 22/22 Top-3.Source: §E.2 · p. 18

F Statistical Tests

  • All five baseline comparisons remain significant at p < 0.05 after correction, using paired MRR bootstrap and Holm-corrected exact McNemar tests.The results are reported in Table 3.Source: F Statistical Tests · p. 18

F.1 Family-level dependence sensitivity

  • Family-clustered bootstrap intervals for BReF’s MRR advantage over all five baselines stay above zero, and every leave-one-family-out evaluation preserves a positive advantage.These are conservative sensitivity checks because only six family clusters are available.Source: §F.1 · p. 19

G Reproducibility Checklist

  • The experiment archive includes benchmark and provenance mappings, frozen probes, cached scores and ranks, analyses, scripts, logs, and a SHA256 manifest; validation covers 411 pairs and 34 repositories.Validation also confirms 22 suspects and 22 provenance edges.Source: G Reproducibility Checklist · p. 19

H Ethical Considerations

  • Behavioral similarity can support provenance audits but also risks vendor de-anonymization or unsupported accusations; attribution should be combined with documentation and human review.The authors state that BReF is technical evidence, not standalone proof of ownership or misconduct.Source: H Ethical Considerations · p. 19
Loading 2609.06330v1…