PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Leaves Traces: Centered Residual Signatures for Language Model Lineage Verification

Aman Singh Thakur, Rayan Khoury

arXiv:2608.14929v1cs.CLcs.LG

TL;DR

Open-weight models undergo transformations that create undocumented descendants, making it difficult to distinguish true ancestry from behavioral similarity. This paper removes shared identity-aligned structure from residual branch products and compares checkpoint-specific signatures, achieving perfect separation on controlled benchmarks while remaining stable under laundering.

  • Problem

    Undocumented model transformations make it difficult to distinguish true weight descendants from models that merely behave similarly.

  • Method

    The method centers residual branch-product signatures, removes shared identity-aligned structure, and calibrates a symmetric lineage score against independent checkpoints.

  • Results

    AUROC=1.0 with perfect separation on GPT-2 benchmarks, while the score remains stable under function-preserving laundering and separates descendants from independent and distilled models.

  • Takeaways & Limitations

    The results support passive, data-free provenance auditing that distinguishes weight ancestry from behavioral similarity across tested open-weight checkpoints.

  • Takeaways & Limitations

    Verification is limited to compatible white-box residual checkpoints with matching depth and hidden dimension, excluding API-only and cross-architecture comparisons.

Abstract

from arXiv · show

Open-weight language models are fine-tuned, quantized, pruned, and merged, yet their provenance is often undocumented. We study data-free white-box lineage verification: can weights alone reveal whether two compatible model checkpoints share ancestry? Residual training produces a shared identity-aligned component in branch products, so this structure alone cannot establish ancestry. We remove it and compare checkpoint-specific structure across residual blocks, yielding a symmetric lineage score calibrated against independent checkpoints. On residual-MLP and GPT-2 benchmarks, the score separates fine-tuned, LoRA-merged, pruned, and quantized descendants from independent and distilled models (AUROC=1.0), distinguishing weight ancestry from behavioral similarity. Under function-preserving checkpoint laundering experiments, weight-space baselines lose margin or fail; our score remains unchanged and runs 76x faster than the nearest robust baseline on GPT-2. The projection-pairing signal appears across six language-model families and beyond, and a case study correctly identifies 3 related and 7 unrelated LLaMA-2 public checkpoints. Collectively, these results establish a passive, data-free provenance signal for compatible open-weight language-model checkpoints

1 Introduction

The paper frames data-free, white-box lineage verification as distinguishing shared weight ancestry from behavioral similarity in compatible open-weight checkpoints. It introduces centered residual signatures that remove generic identity-aligned structure and compare checkpoint-specific remainders, with calibrated scoring and broad empirical validation.

  • Introduction: Open-weight models undergo fine-tuning, quantization, pruning, low-rank adaptation, merging, and redistribution, often obscuring their undocumented ancestry.These operations create weight descendants that may be renamed or intentionally obscured.
  • Introduction: The task is to determine from weights alone whether compatible residual-architecture checkpoints share a common weight ancestor, even when unrelated models behave similarly.Relatedness includes inheritance through fine-tuning, RLHF, pruning, quantization, or LoRA merging; independent initialization defines unrelatedness.
  • Introduction: Centered residual signatures remove generic identity-aligned branch-product structure and compare checkpoint-specific remainders across blocks using a symmetric score calibrated against independent checkpoints.The generic structure appears in independently trained residual models, so it cannot establish lineage by itself.
  • Introduction: 100% accuracy was achieved for Hungarian recovery of canonical MLP projection pairings across all tested families, while the prerequisite trace-concentration signal appeared across six language-model families.A LLaMA-2 case study compared 3 documented descendants with 7 independently trained architectural clones.
  • Introduction: The model-level score uses empirical-null calibration and is invariant to hidden-unit permutation and reciprocal rescaling.These properties are presented as core contributions of the proposed score.
  • Introduction: The evaluation tests post-training transformations, checkpoint laundering, distillation, and projection-pair recovery across six language-model families.These experiments assess robustness and distinguish lineage from behavioral or architectural similarity.

2 Problem Formulation

The problem is to verify weight-level lineage between two compatible checkpoints using only their weights, distinguishing shared ancestry from matching architecture, data, tasks, or outputs. Existing alternatives either require proactive signals, retained training artifacts, probe data, or only measure functional similarity rather than weight inheritance.

  • 2.1 White-Box Model Lineage Verification: The verifier receives checkpoints A and B and returns RELATED or UNRELATED for weight-level lineage.It assumes white-box access and compatible architectures with the same L and d.
  • 2.1 White-Box Model Lineage Verification: The method uses weights alone, without training data, activations, or forward passes.Cross-architecture comparisons are out of scope.
  • 2.1 White-Box Model Lineage Verification: Related checkpoints share a common weight ancestor through post-training, whereas independently initialized or distilled models are unrelated even when outputs match.Linear merges induce partial lineage.
  • 2.1 White-Box Model Lineage Verification: The lineage score is symmetric, L(A, B) = L(B, A), while directional attribution requires metadata.The verifier establishes relation rather than direction.
  • 2.2 Related Work: Existing provenance mechanisms fail under checkpoint redistribution or modification, or require proactive insertion and retained training records.Hashes break after weight changes; model cards depend on honest distributors; proof-of-learning needs training transcripts; watermarks require insertion during training.
  • 2.2 Related Work: Functional and representational similarity methods do not by themselves certify weight inheritance.IPGuard and adversarial frontier stitching use decision-boundary inputs, while CKA and SVCCA use probe datasets.
  • 2.2 Related Work: Weight-space baselines detect derived checkpoints only when they remain close in parameter space and provide uncalibrated global scalars.Other approaches require parameter convergence, probe data, activations, or model-tree recovery, whereas the fingerprint uses a reference-specific empirical null and per-block scores.
  • 2.2 Related Work: Trace concentration in correctly paired residual-block projections appears across six language-model families but cannot alone establish lineage.The paper therefore separates identity-aligned from checkpoint-specific components to construct a model-level lineage score.

3 Residual Signatures for Lineage Verification

The method removes the generic identity-aligned component from residual branch products and compares centered, checkpoint-specific traceless signatures across aligned blocks. Empirical tests show this signal reflects residual training dynamics, survives weight-preserving transformations, and supports calibrated lineage verification rather than merely detecting trained structure.

  • Centered residual signatures: Branch products decompose into an identity component and an orthogonal traceless remainder, with the verifier retaining the centered remainder for checkpoint-specific comparison.The identity coefficient is tr(M_ℓ)/d under the Frobenius inner product, while tr(E_ℓ) = 0.
  • Cross-family projection pairing: 100% block-pairing accuracy is achieved across six language-model families on the canonical MLP path, versus random-initialization baselines of at most 4%.Alternative factorizations show more variation, while Hungarian matching recovers within-block projection correspondences.
  • Training mechanism: 68% of the fingerprint is lost when gradients are shuffled across blocks, while synthetic identity-aligned updates build it from scratch without backpropagation.Gradient diagonality remains approximately 0.15 as the weight score rises to approximately 4.0, indicating accumulation of correlated updates rather than diagonal individual updates.
  • Checkpoint specificity: 18× improvement reduces spurious independent-model cosine similarity from 0.027 uncentered to 0.0015 after centering on the GPT-2 benchmark.The centered traceless remainder is checkpoint-specific, survives weight-preserving transformations, and is not reproduced by independent training.
  • Calibrated lineage verification: AUROC=0.417 is obtained by a trace-only verifier, whereas lineage verification compares centered signatures with a symmetric score calibrated against independently trained-model nulls.Related checkpoints score near 1 and unrelated checkpoints near 0; RELATED is returned when the observed score exceeds the maximum null score.

4 Experiments

Across controlled, public-checkpoint, and laundering experiments, the centered residual signature separates weight-ancestral descendants from independent or distilled models. It remains robust to function-preserving transformations while requiring substantially less computation than alignment-based comparison.

  • Controlled benchmarks: On MLP, four weight-space methods—including ours—achieve AUROC=1.0, while CKA and IPGuard underperform at AUROC=0.83 and 0.70.SVCCA also achieves perfect discrimination but requires forward passes through both models.
  • Controlled benchmarks: On GPT-2, ours, weight cosine, and aligned Frobenius maintain AUROC=1.0, while singular-value distance degrades to AUROC=0.73.SVCCA remains strong at 0.99, whereas CKA reaches 0.86 and IPGuard 0.91.
  • Public checkpoints: The LLaMA-2 case study gives documented derivatives scores of L: 0.995, 0.996, and 0.336, while all 7 independent models score |L| < 5 × 10−5.The comparison uses identical-architecture checkpoints: 32 layers, d=4096, intermediate=11008, and SwiGLU.
  • Function-preserving laundering: Under function-preserving laundering, our signature and Re-Basin+scale maintain AUROC=1.0 across all conditions, while raw baselines collapse.Aligned Frobenius drops to 0.50 under permutation and 0.0 under strong rescaling; weight cosine reaches 0.80 under PD.
  • Function-preserving laundering: 76× speedup on GPT-2 and 2× speedup on MLP result from direct branch products, with runtimes of 5ms versus 388ms and 0.4ms versus 0.8ms.The method computes O(Ld^2h), avoiding Re-Basin+scale’s O(Ld^3) Hungarian alignment search.
  • Transformation robustness: On GPT-2, quantization and LoRA preserve lineage scores at L≥0.996, while fine-tuning and pruning remain above the null at L≥0.855.Distilled students and independently trained models score near zero, producing AUROC=1.0 from a 200× descendant-to-non-descendant gap.

5 Conclusion

The paper introduces a data-free lineage detection score based on checkpoint-specific structure in residual branch products. Across benchmarks and model families, it separates weight ancestry from behavioral similarity, remains stable under function-preserving checkpoint laundering, and matches the nearest robust baseline at lower latency.

  • Conclusion: The proposed data-free score derives lineage evidence from checkpoint-specific structure in residual branch products.It is presented as a lineage detection score rather than a behavioral-similarity measure.
  • Conclusion: Across residual-MLP and GPT-2 benchmarks, the score separates fine-tuned, LoRA-merged, pruned, and quantized descendants from independently trained and distilled models.This separation distinguishes weight ancestry from behavioral similarity.
  • Conclusion: The score remains stable under function-preserving checkpoint laundering and matches the nearest robust baseline at lower latency.Projection-pairing results also appear across six language-model families.

Ethical Considerations

The method supports provenance verification and supply-chain auditing but produces similarity scores rather than legal determinations. Responsible interpretation requires metadata, sufficient calibration, consent-aware use, human review, and appropriate evidentiary standards.

  • Interpretation limits: Similarity scores do not independently prove provenance, ownership, or wrongdoing and should be combined with metadata, release records, and human review.The method detects shared weight ancestry but does not make legal determinations.
  • Calibration limits: Limited calibration data, especially few independent roots, may create spuriously precise thresholds for rare architectures.Formal hypothesis testing requires sufficient exchangeable samples, which may be unavailable.
  • Privacy and consent: White-box access limits covert use, but authorized access does not automatically imply consent to provenance analysis.The method may identify undocumented relationships that conflict with developers’ reasonable expectations of anonymity.
  • Misuse risks: The method could be misused for unfounded infringement claims or competitive intelligence despite legitimate supply-chain auditing applications.The authors encourage responsible deployment with appropriate evidentiary standards.

Limitations

The method requires compatible white-box residual checkpoints and external calibration, while offering symmetric pairwise evidence rather than directional or multi-hop ancestry reconstruction. Its signal weakens under aggressive transformations and is not universally invariant or robust beyond the evaluated settings.

  • Scope: Verification requires white-box access to both checkpoints, residual architectures, and matching depth and hidden dimension.API-only models, plain feedforward networks, RNNs, state-space models, and cross-architecture comparisons are out of scope.
  • Interpretation: The symmetric score cannot identify the ancestor, reconstruct multi-hop ancestry, or identify which blocks were inherited under partial inheritance.Without external metadata, it only provides pairwise evidence; under layer grafts and linear merges, it reflects the fraction of shared blocks.
  • Robustness: L = 0.58 at 85% sparsity and L = 0.336 after extensive CodeLlama continued pretraining, though both remain above the null.In the evaluated MLP suppression attack, reaching L ≈0.084 costs +1.5% utility loss, while reliably driving the score below null costs +12%.
  • Invariances: The signature is invariant to hidden-unit permutation and reciprocal MLP rescaling but is destroyed by orthogonal residual-stream rotation.Rotation preserves trace but transforms the centered remainder; it is not function-preserving for LayerNorm architectures with learned element-wise γ and β parameters.
  • Calibration: Perfect AUROC discrimination was demonstrated in controlled benchmarks, but calibration requires same-architecture independent models and formal-testing sample sizes remain limited.The GPT-2-style benchmark has 8 roots total, while the LLaMA-2 case study includes 7 independent models; thresholds may need adjustment across architecture families.

Use of AI Assistants

The authors retain full responsibility for the work’s scientific claims, results, and analysis, while using Claude during manuscript preparation with all AI-assisted material subsequently reviewed, verified, and revised.

  • The authors bear full responsibility for all scientific claims, experimental results, and analysis.
  • Claude supported literature searches, experimental design, code development, and manuscript drafting and editing; the authors reviewed, verified, and revised all AI-assisted material.

A Reproducibility Statement … C.2 GPT-2 Scaling, Trace, and Jacobian Analysis

The paper reports deterministic, reproducible benchmarks and implementation details, while validating architecture-aware lineage signals across latency, factorization, calibration, initialization, and GPT-2 mechanism analyses. Results show strong pairing performance, substantial speed advantages, and evidence that trace concentration is not explained by near-isometric Jacobians.

  • A Reproducibility Statement: Deterministic scripts, fixed seeds, specified benchmark constructions, and a single L4-based software stack support reproducible evaluation.The MLP benchmark uses 52 pairs from two trained references, while GPT-2-Small-Lite uses eight roots with held-out test splits.
  • A.1 Latency Statistics: 76× faster than Re-Basin+scale and 283× faster than SVD, the method achieves identical AUROC under laundering.These latency results are reported for 20 GPT-2 benchmark pairs evaluated on GPU.
  • A.2 Architecture-Aware Factorization: 100% accuracy with AUC 1.000, 100% with AUC 0.995, and 91% with AUC 0.964 are recovered by correct triple products on ResNet-50, -101, and -152.Naive two-layer factorization achieves chance-level accuracy, whereas architecture-matched W3W2W1 products recover the signal.
  • B.1 Gated Branch Score: Gated branch similarity is used only when both branches have strong trace concentration, with the gate set by minimum reference-branch concentration.This condition defines the gated branch score before statistical calibration.
  • A.3 Per-Path Pairing Accuracy: The canonical MLP down×up path achieves 100% pairing accuracy across tested language-model families, while some down×gate paths are weaker.Qwen reaches 68% and DeepSeek 84% for the alternative SwiGLU factorization.
  • B.2 Statistical Calibration: All 75 related pairs exceed max Lnull = 0.20 against 84 unrelated pairs, yielding AUROC=1.0 under the reported lineage-score calibration.The effective calibration sample size is limited by independent roots rather than descendant-pair counts.
  • C.1 Initialization Ablation: After training, initialization-scheme pairing accuracy rises to 93–100% from chance-level accuracy of ≤2.1% at initialization.This includes orthogonal initialization, which has dynamical isometry at initialization but zero correct pairs before training.
  • C.2 GPT-2 Scaling, Trace, and Jacobian Analysis: Pretrained GPT-2 models are 5–12× less orthogonal than random initialization, contradicting the hypothesis that trace concentration comes from near-isometric blocks.The analysis spans GPT-2 scales from 124M to 1.5B parameters and examines pairing scores and trace values across variants.

C.3 Centering Ablation · D Extended Benchmarks

Centering removes the identity-aligned component shared across trained models, exposing checkpoint-specific residual structure. On GPT-2, centering preserves perfect separation while substantially reducing spurious similarity, especially between independent roots.

  • C.3 Centering Ablation: Centering removes the identity-aligned component shared by all trained models before comparing branch products.This exposes the checkpoint-specific remainder that differs between unrelated models.
  • C.3 Centering Ablation: AUROC=1.0 for both centered and uncentered methods on the GPT-2 benchmark.The benchmark contains 27 pairs from 3 test roots.
  • C.3 Centering Ablation: ∼50× lower unrelated-pair similarity after centering, decreasing from 0.019 to 0.0004.This reduction removes spurious similarity without changing perfect related-versus-unrelated separation.
  • C.3 Centering Ablation: 18× lower average similarity for independent roots after centering, decreasing from 0.027 to 0.0015.Independent-root similarity averages 0.027 uncentered and 0.0015 centered.
  • C.3 Centering Ablation: F averages 9.4 across all layers, indicating that the identity component carries substantial matrix energy.Independent models share this generic identity-aligned structure from training dynamics.
  • C.3 Centering Ablation: Centering reveals checkpoint-specific structure by removing generic identity-aligned structure shared by independent models.The resulting remainder differs between unrelated checkpoints.

D.1 Laundering Experiment Gap-Z Scores … D.6 Gradient-Based Suppression Attack

Across laundering, partial-overlap, vision, language-model, and attack settings, the centered residual signature preserves lineage separation where simpler weight-space signals weaken or fail. It also distinguishes weight ancestry from behavioral similarity and quantifies the utility cost of suppressing the signal.

  • D.1 Laundering Experiment Gap-Z Scores: Gap-Z remains ≈+53 under all laundering conditions, while weight cosine collapses from +76.3 to +2.1 under permutation and +1.8 under PD.Weight cosine retains AUROC=0.80–0.86 despite nearly overlapping descendant and non-descendant distributions.
  • D.2 Harder Regime: Layer Grafts and Linear Merges: The harder 40-pair graft-and-merge benchmark tests partial overlap using block-copy counts K and merge weights w, with relatedness thresholds K≥8 and w≥0.5.AUROC measures relatedness detection, while Spearman correlation tests monotonic tracking of overlap.
  • D.2 Harder Regime: Layer Grafts and Linear Merges: Our method tracks partial overlap monotonically with ρ ≥0.96, whereas CKA degrades, IPGuard collapses on grafts, and SVD fails on merges.The benchmark contains 40 pairs across layer grafts and linear merges.
  • D.3 CIFAR-10 ResNet-18 Benchmark: AUROC=1.000 on CIFAR-10 ResNet-18, with related scores L ∈[0.695, 1.000] and an unrelated baseline of Lmax = 0.004.Each reference produces 11 related checkpoints through noise, pruning, and quantization transformations.
  • D.4 GPT-2-Small-Lite Benchmark: In GPT-2-Small-Lite, the minimum descendant score 0.855 exceeds the maximum non-descendant score 0.004 across quantized, LoRA, fine-tuned, and pruned descendants.Distillation raises teacher–student agreement to 79% top-1, yet its weight-lineage score remains near the independent-root range.
  • D.5 ROC for Lineage Verification: On depth-24 residual MLPs, the residual-signature score achieves AUROC=1.000, while trace concentration alone fails with AUROC=0.417.Trace concentration cannot distinguish two trained residual models.
  • D.6 Gradient-Based Suppression Attack: Suppressing the lineage score to the null baseline Lnull = 0.084 costs +1.5% utility loss, while reliably driving below null costs +12%.The attack optimizes perturbation ∆ subject to utility constraints on depth-24 MLPs with d=64.

E Transfer Beyond Language Models

The trace concentration signal extends beyond language models to vision and speech architectures, where architecture-aware branch products recover projection pairings. In ResNets, using the correct three-layer bottleneck product recovers 91–100% of layer3 block pairings, while naive two-layer factorization reaches chance level.

  • Cross-architecture generalization: Architecture-aware branch products recover projection pairings in both vision and speech architectures, demonstrating that trace concentration is not specific to language models.The evaluation includes ViT, Whisper, and ResNet architectures.
  • Cross-architecture generalization: The cross-architecture setup applies the language-model transformer MLP factorization W2W1 to ViT and extracts branch products from both encoder and decoder MLP blocks in Whisper.These choices adapt branch-product extraction to each architecture's block structure.
  • Cross-architecture generalization: 91–100% of layer3 block pairings are recovered in ImageNet-pretrained ResNet-50/101/152 using the correct triple product W3W2W1.ResNet bottleneck blocks require a three-layer product rather than a two-layer factorization.
  • Cross-architecture generalization: Naive two-layer factorization W3W1, which skips the middle convolution, yields chance-level pairing accuracy on ResNet bottleneck blocks.The correct triple product is necessary for reliable projection recovery in these blocks.
Loading 2608.14929v1…